Scenario Based User Testing

The most effective method of user testing is to conduct scenario based tests in facilitated sessions. The following paragraphs describe this methodology.

Identifying the Scenarios

The first step is to identify the most important tasks or usage scenarios. In addition to the homepage, key scenarios on most websites usually include locating and using the sitemap, contact page and search facility. Other typical scenarios include registration, the login process and completing a purchase.

In some cases the user will be instructed to test a specific page. However, the process of locating a page or functional area must be as accessible as the page or function itself so they will usually be given a task and left to complete it, starting from the homepage or another appropriate starting point.

Facilitated Sessions

The user works in conjunction with a facilitator. The user will not have seen the site previously, but will have been briefed on its content and purpose. The facilitator has detailed knowledge of the site but is careful not to influence the user's behaviour.

The user works through each scenario in turn, describing their approach, reasoning and expectations, which is known as a 'think aloud protocol'. Meanwhile the facilitator takes notes and prompts the user where more detail is required. The sessions may be video recorded for further analysis if required.

Facilitated sessions capture a great deal of important information that is lost when other methodologies are used. Although it would be cheaper for the user to work alone we do not recommend it for the following reasons.

  • The user cannot comment on what they cannot see, and it is not uncommon for some content to be invisible to screen readers. Also users may be unaware of the presence of some content or functionality or may misinterpret its meaning or function.

    If the user is working alone they will be unable to report these issues, but they will be obvious to a skilled facilitator.
  • The facilitator can adapt and extend the test scenario while it is in progress in response to the user's behaviour. This allows problems to be investigated in greater detail as they occur.

    In particular it allows information to be gathered after the user encounters a problem that would otherwise be a showstopper.
  • First impressions are very important, as is observation of the process by which the user resolves any difficulties they experience. The facilitator can capture this information accurately in real time, whereas the process of keeping notes is disruptive for a user working alone, potentially invalidating the results.

Interpreting the Results

When interpreting the results it is important to take into account factors including the type, severity and history of the user's disability, the type of assistive technology they use, their level of computer literacy, domain knowledge and possibly demographic factors such as their age and occupation.