Jakob Nielsen
Thomas K. Landauer
ABSTRACT
For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.
- 1.Dalal, S.R., and Mallows, C.L. (1988). When should one stop testing software? J. American Statistical Association 83, 403 (September), 872-879.Google Scholar
Cross Ref
- 2.Dalai, S.R., and Mallows, C.L. (1990). Some graphical aids for deciding when to stop testing software. IEEE J. Selected Areas in Communication 8, 2 (February), 169- 175.Google Scholar
- 3.Erhan, S. (1975). Introduction to Stochastic Processes. Prentice Hall, Englewood Cliffs, NJ. p. 87.Google Scholar
- 4.Gray, W.D., John, B.E., and Atwood, M.E. (1992). The precis of project Ernestine, or, an overview of a validation of GOMS. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 307-312. Google ScholarDigital Library
- 5.Jeffries, R., Miller, J.R., Wharton, C., and Uyeda, K.M. (1991 ). User interface evaluation in the real world: A comparison of four techniques. Proc. ACM CHI'91 (New Orleans, LA, 27 April-2 May), 119-124. Google ScholarDigital Library
- 6.Jq~rgensen, A.H. (1989). Using the thinking-aloud method in system development. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Amsterdam: Elsevier Science Publishers, 743-750. Google ScholarDigital Library
- 7.Karat, C., Campbell, R., Fiegel, T. (1992). Comparisons of empirical testing and walkthrough methods in user interface evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 397-404. Google ScholarDigital Library
- 8.Lewis, C. (1982). Using the 'thinking-aloud' method in cognitive interface design. Research Report RC-9265, IBM T.J. Watson Research Center, Yorktown Heights, NY.Google Scholar
- 9.Lewis, J.R., Henry, S.C., and Mack, R.L. (I 990). Integrated office software benchmarks: A case study. Proc. INTERACT'90 3rd IFIP Conf. Human-Computer Interaction (Cambridge, U.K., 27-31 August 1990), 337-343. Google ScholarDigital Library
- 10.Mantei, M.M., and Teorey, T.J. (1988). Cost/benefit analysis for incorporating human factors in the software lifecycle. Communications of the ACM 31, 4 (April), 428-439. Google ScholarDigital Library
- 11.Molich, R., and Nielsen, J. (1990). Improving a humancomputer dialogue. Communications of the A CM 33, 3 (March), 338-348. Google ScholarDigital Library
- 12.Nielsen, J. (1989). Usability engineering at a discount. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems, Elsevier Science Publishers, Amsterdam. 394-401. Google ScholarDigital Library
- 13.Nielsen, J. (1990). Big paybacks from 'discount' usability engineering. IEEE Software 7, 3 (May), 107-108. Google ScholarDigital Library
- 14.Nielsen, J. (1992). Evaluating the thinking aloud technique for use by computer scientists. In Hartson, H.R., and Hix, D. (Eds.), Advances in Human--Computer Interaction Vol. 3, Ablex. 69-82. Google ScholarDigital Library
- 15.Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 373-380. Google ScholarDigital Library
- 16.Nielsen, J. (1992). The usability engineering lifecycle. IEEE Computer 25, 3 (March), 12-22. Google ScholarDigital Library
- 17.Nielsen, J. (1993). Estimating the number of subjects needed for a thinking aloud test. Intl. J. Man-Machine Studies in press.Google Scholar
- 18.Nielsen, J. (1993). Usability Engineering, Academic Press, San Diego, CA. Google ScholarCross Ref
- 19.Nielsen, J. (1993). Iterative design of user interfaces. IEEE Computer 26 (to appear, probably in the July issue). Google ScholarDigital Library
- 20.Nielsen, J. (1993). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. Book in preparation. Google ScholarDigital Library
- 21.Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 (Seattle, WA, 1-5 April), 249-256. " Google ScholarDigital Library
- 22.Virzi, R.A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting (Orlando, FL, 8-12 October), 291-294.Google ScholarCross Ref
- 23.Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects are enough? Human Factors 34, 4 (August), 457-468. Google ScholarDigital Library
- 24.Wright, P.C., and Monk, A.F. (1991). A cost-effective evaluation method for use by designers. Intl. J. Man- Machine Studies 35, 6 (December), 891-912. Google ScholarDigital Library
Index Terms
- A mathematical model of the finding of usability problems
Recommendations
Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared
CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing SystemsTwo alternative user interface designs were subjected to user testing to measure user performance in a database query task. User performance was also estimated heuristically in three different ways and by use of formal GOMS modelling. The estimated ...
Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsUsability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
Usability in practice: formative usability evaluations - evolution and revolution
CHI EA '02: CHI '02 Extended Abstracts on Human Factors in Computing SystemsFormative evaluation is a collection of "find-and-fix" usability engineering methods, focused on identifying usability problems before a product is completed. In this forum, four experienced usability professionals will address different aspects of ...
Comments