by Scott McDaniel
Reprinted from Usability Interface, Vol 6, No. 1, July 1999
The problem: How do you conduct usability tests when your users are scattered across the country and there is no possibility of site visits?
One solution: Create a tester’s guide to send to surrogate testers who, though untrained, perform the usability test for you.
How the Strategy Works
I recently used this strategy to test an online help database I was building in Lotus Notes. The system that I am documenting is a suite of Lotus Notes programs that allow organizations to apply to the Environmental Protection Agency (EPA) for grant funds. The system helps a user create and track application materials, forward them to others for review, and eventually submit them. EPA uses the system to create and process award documents based on the applications. Both the applicant users and EPA users are all over the country, and though my management approved the usability testing, they did not allocate money for site visits or equipment.
EPA is implementing this application and award system as a pilot project, with iterative design and direct contact between developers and users. A pilot community was therefore already established, and I contacted four of the participants to see if they would conduct a usability test with another user at their site.
I prepared a tester’s guide to send to them, with complete instructions for conducting the test. Once they had reviewed the tester’s guide and the prototype of the online help, I contacted them by phone to go over the procedures and give them an opportunity to ask questions.
After they conducted the tests, they sent the results back to me. I compiled the data and observations into a test report, which I then shared with both testers and management. The guide itself contained three sections. The first oriented the tester to the process, providing step-by-step instructions on how to proceed, plus a list of needed materials and a sheet for demographic data about the user.
The test procedure required participants to work in pairs: one as the user who performed the tasks and one as the test administrator (tester). The second section contained 10 questions that users might have as they performed their tasks. The user was to look up the help topic answering the question while the tester recorded the time needed to find the correct topic, as well as any errors. After each task, the tester asked the user about her search strategy and took notes. The final section of the tester’s guide provided space for general user and tester comments.
What to Watch For
Using surrogate testers is no substitute for being there yourself, but it is far better than no testing at all. If you decide to use surrogate testing, keep in mind the issues below.
Some testers took liberties with the administration of the test. For example, one decided that two of the 10 tasks did not apply to his situation, and so he omitted them. Another tester gave the user a little more help than I specified in the testing instructions. These variations gave clues as to the user and tester views of their jobs, but introduced all the validity questions that arise when you don’t hold a study’s method constant. Relying on surrogate testers delayed the usability process. I had to repeatedly contact the testers to remind them to conduct the tests and send the results to me. Setting up phone meetings where I could talk to them or debrief them was sometimes difficult.
There was a great deal of telephone and e-mail tag. The testers have demanding jobs without their doing this favor for me, and so I did not want to badger them. Having different testers introduced variability into the data recording. Some testers recorded more detail, others less. One tester/user’s times was much longer than the others, leading me to think that he did not record times the same way that others did. In spite of these difficulties, the test turned up clear usability issues. While the hard numbers are suspect, I have always found greater value in the user comments and misunderstandings than in time reports. For example, two questions out of the ten caused difficulties for every subject. In both cases, the comments revealed a mismatch between my vocabulary (as the online help author) and the users’ vocabularies. I was able to modify the language in the help topics to match the users’ conceptions. In another example, none of the users realized that they could look up help topics multiple ways. They always used the default method.
What I Learned
Should you decide to use the surrogate tester approach, keep the following recommendations in mind:
- Make the tester’s guide detailed, and emphasize the importance of adhering to its procedures.
- Go over the guide with the testers before they conduct the test. A phone call allows the tester to ask questions and verify assumptions.
- Usability test your tester’s guide. (Caroline Jarrett suggested this in a personal correspondence.)
- Build extra time into your schedule to accommodate the difficulties in contacting all of your testers both before and after the tests.
- Provide clear motivation for your tester to complete the test and return it to you. The motivation may be internal (an opportunity to help develop the new system) or external (money, free technical support, and so on).
