Agent Wranglers

UXD-9Recommended

Test each main task with five users before launch

Before a launch or a redesign, watch at least five people from the target group try each main task without help. Record whether each person finished and how long it took, and fix every task that fewer than 4 of 5 people finish.

By Agent Wranglers. Updated .

How to do it

The team that built a screen cannot see what confuses a newcomer. Write each main task as a goal, such as "change the delivery address of order 1042", never as a path through the menus. Give no hints during the session. Recruit from the people who will use the product, and include disabled people, as Test with disabled people says. Run the same tasks again after the fix, so the report shows the change from the first round. Test the menu structure first, as Tree test the navigation before you build it says.

Examples

TypeScript example

const PASS_SHARE = 0.8

type Attempt = { task: string; finished: boolean; seconds: number }

export function tasksToFix(attempts: Attempt[]): string[] {
  const byTask = Map.groupBy(attempts, attempt => attempt.task)
  return [...byTask].filter(([, tries]) => tries.filter(t => t.finished).length / tries.length < PASS_SHARE).map(([task]) => task)
}

Python example

from collections import defaultdict

PASS_SHARE = 0.8

def tasks_to_fix(attempts: list[dict]) -> list[str]:
    by_task: dict[str, list[bool]] = defaultdict(list)
    for attempt in attempts:
        by_task[attempt["task"]].append(attempt["finished"])
    return [task for task, results in by_task.items() if sum(results) / len(results) < PASS_SHARE]

How to check

The launch checklist links a test report dated within 90 days before the launch. It lists every main task with at least five participants, a finished or not finished result and a time for each. Every task that fewer than 4 of 5 finished has a fix and a retest.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/ux-design/research-and-structure/usability-tests">Agent Wranglers UX and deceptive design playbook</a>. In code you write, add a comment next to it with the same link.