By Denise Pierotti, Xerox Corporation
Heuristic Evaluation
Heuristic Evaluation (expert review) is a diagnostic method in which experts take the role of less experienced users and describe the potential problems they see arising in a system or interface for those users. The review is based on compliance with a set of principles (heuristics). It is known as a “discount” method, and was designed for quick turnaround so that the deliverable can be attended to as part of an iterative design process.
References
“Heuristic Evaluation”, in Usability Inspection Methods, edited by J. Nielsen and R. Mack, John Wiley and Sons, Inc., 1994, pages 25-62.
Getting Ready (Project Lead activities)
- Identify and define the usability principles (heuristics) that you will use to evaluate the user interface system.
- Select your evaluation team. Identify three to five usability professionals to examine the system on an individual basis.
- Schedule locations, days, and times for each usability professional to assess the system.
- Prepare or compile materials that will enable the evaluators to become familiar with the purpose of the system and of its users. These materials can include: audience analysis, system specification, user tasks, use case scenarios, etc. Distribute these materials to the evaluators.
- Design your evaluation and notetaking strategy. Are you are going to evaluate the system on an individual or group basis? Are you are going to assign a common notetaker or ask each individual to take his or her own notes?
Evaluating the system (Evaluator activities)
- Experiment with and establish a feel for the scope of the system.
- Review the materials provided to familiarize yourself with the system design. Perform the user actions that you feel would be taken to perform the user tasks.
- Identify and list any areas of the system that you feel are counter to the heuristics. List all of the concerns that you note, including what seem to be duplicates. Be sure to clearly describe what you find, including where in the system it was found.
Analyzing the results (group activity)
- Review each of the concerns noted by each of the evaluators. Make sure that each concern is clearly understood by all evaluators.
- Develop an affinity diagram that groups similar concerns.
- Evaluate and judge each concern for its compliance with your defined heuristic.
- Assign an severity level for each grouped concern based on the impact to the end user(see Severity Rating section).
- Determine recommendations to fix the problem. Make sure each recommendation links the heuristic and a design principle.
Reporting the results (Team Leader activity)
- Compile the results of the group meeting of the evaluators. Each problem should have a severity code, a link to a design principle, an explanation of the usability issue, and a recommendation.
- Report all sources, purposes, techniques, procedures, and findings in a format that is easy to read and understand. You may decide to organize you findings by design principle (heuristic). Be sure to note the positive attributes of the system or interface.
- Make sure the report includes a mechanism for the Project Team Lead to report back on how the information was used by the development team.
- Have your report reviewed by another team member, and approved by team coach.
Debriefing (Team Leader activity)
- Schedule a time and location for your oral presentation if required by the customer.
- Focus on the major usability concerns and possible solutions.
- Highlight the positive attributes concerning the design.
- Follow up with the Project Team Lead as necessary.
Severity Rating Scales
You can decide to use either of these severity rating scale, or develop you own to suit the project needs.
Five-point rating scale
1 Cosmetic, will not affect the usability of the system, fix if possible.
2 Minor, users can easily work around the problem, fixing this should be given low priority.
3 Medium, users stumble over the problem, but quickly adapt to it, fixing this should be given medium priority
4 Major, users have difficulty, but are able to find workarounds, fixing this should be mandatory before the system is launched. If the problem cannot be fixed before launch, ensure that the documentation clearly shows the user a workaround
5 Catastrophic, users are unable to do their work, fixing this is mandatory
Three-point scale
- Low cosmetic or minor, causes minimal difficulty
- Moderate causes some problems to doing work or causes the user to stumble, but recovery is possible.
- High effectively prevents the user from doing work, the user will fail or have extreme difficulty.
