EDC user testing guide (Carmichael)
User Testing Guide
Carmichael/Sabo/Birnbaum WQ 2008
Remember: Although this process is called “User Testing,” you are testing your design alternatives, NOT the user(s) or their intelligence or ability levels. If something goes wrong, the design is flawed—not your user.
Planning the User Test
After you’ve identified users and have set up a time to meet with them, you’ll need to plan the user testing of your mockups. The most important element in the planning phase is to set an objective(s) for your test and then design a task that will allow you to evaluate a specific aspect of the design that you are testing. This will help you limit the scope of the test and gather specific information. The tests that you develop should include activities that users would ordinarily perform with your design (based on your research and client information).
The Interview and Observation Guide
After you have determined what you are testing and how, specifically, you will test it, your team will need to create an interview and observation guide to ensure consistency in your testing across team members and across users. Here are some key elements of your interview guide and how the testing should proceed:
1) Introduction: Introduce yourselves and the purpose of the testing session in very general terms (e.g. you are engineering students developing a new design /modifying an existing design and would like them to try a few design mockups). AVOID mentioning your specific objectives—such information may affect their interaction with your design. Make clear the following:
a. You’re testing your design and not them.
b. That thinking “aloud” is encouraged: i.e. tell them that it’s very helpful to them if they can describe what they’re thinking or feeling as they approach, evaluate, or try to use your design. (You may need to reassure them that you’re thick-skinned and not wedded to anything at this point).
c. Ask if you can take video or photos. If the user agrees, ask if the user wishes to be identified or kept anonymous in those videos or pictures. In the event of confusion on this point, always default to anonymity.
2) Demographic Questions: Come prepared with a set of questions for your users that will help you categorize them and their experience and knowledge of your design. Remember, you will be ideally testing your design with multiple users—it’s helpful to capture a snapshot of who these users are in writing.
3) Task Set Up: Describe in general terms the various tasks you’d like each user to perform. Your instructions should be complete, but should not “give away” what kind of information you seek in your test (e.g. Don’t tell a user that you are seeing how long it will take them to don a device.) Again: resist the urge to over-explain the design. (See EDC textbook 77 for specific examples.)
4) Task Observation: Allow the participants to perform the tasks without leaping in to rescue them (or the design!) at the first sign of trouble. Remember that you will lose good information if you jump in too quickly. Have at least one person (or preferably two) take detailed notes about what is happening. Better: have someone take notes and someone record on video (if you were given permission).
You can, however, answer questions that the user may ask (e.g. if the user becomes frustrated with the design and the test session grinds to a halt). Someone on the team, however, should be recording all of those questions that come up in user testing. These will give you a sense of where the design’s use is less than intuitive.
a. Remember your role: You are testing your design and are trying to find out what might be difficult or ineffective for your user. Do not try to defend or over-explain your ideas or your favorite solution to your user!
b. Watch for patterns: If testing multiple users, you may find that some with certain characteristics interact with the design in a particular way.
c. Be aware of user preferences: Look for moments where the user may show an aversion to a particular element of a design solution. For example, a stroke survivor may be able (if necessary) to raise his or her arms above the head to use a particular design, but may prefer not to do so. So while your proposed design may represent a possible solution to the problem, it still may not be a preferred solution to the problem.
5) Debrief the Tasks with Users. Prepare questions to debrief the users on their experience. When possible, give users a rating scale that allows them to rank your solutions or a particular feature. Have someone on your team take detailed notes on their comments. (See 89 for examples.)
In addition, remember that the testing context is LIMITED. So, for example, if you are conducting user testing in a clinical setting, you might want to include follow-up questions for the user on how the user imagines your design might work in his or her home (if such a question is applicable and appropriate).
6) Conclude the Interview. Tell your users after the fact what you were testing for specifically and answer their questions.
Be sure to thank them for their time!!
