Saul Greenberg
Bill Buxton
ABSTRACT
Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done 'by rule' rather than 'by thought'. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design's scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today's reluctant reactions by users will forestall tomorrow's eager acceptance. The choice of evaluation methodology - if any - must arise from and be appropriate for the actual problem or research question under consideration.
Supplemental Material
Available for Download
Slides from the presentation
Slides from the presentation
Slides from the presentation
Supplemental material for Usability evaluation considered harmful (some of the time)
- Barkhuus, L., Rode, J. From Mice to Men - 24 Years of Evaluation in CHI. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/ (2007). Google Scholar
Digital Library
- Bush, V. As We May Think. Atlantic Monthly, (1945)Google ScholarDigital Library
- Buxton, B. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann, (2007). Google ScholarDigital Library
- Buxton, W. & Sniderman, R. Iteration in the Design of the Human-Computer Interface. Proc 13th. Meeting, Human Factors Assoc. of Canada, (1980), 72--81.Google Scholar
- Christensen, C. The Innovator's Dilemma. Harper Business School Press, (1997).Google ScholarDigital Library
- Cockton, G. Make Evaluation Poverty History. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/. (2007) Google ScholarDigital Library
- Dix, A., Finlay, J., Abowd, G. and Beale, R. Human Computer Interaction, 2nd Edition, Prentice Hall, (1993) Google ScholarDigital Library
- Dijkstra, E. Go To Statement Considered Harmful. Comm. ACM 11(3): 147--148, (1968). Google ScholarDigital Library
- Edwards, W. and Grinter, R. At Home with Ubiquitous Computing: Seven Challenges. Proc UBICOMP. LNCS 2201, Springer-Verlag, (2001), 256--272. Google ScholarDigital Library
- Engelbart, D.C. and English, W.K. A Research Center for Augmenting Human Intellect. AFIPS Fall Joint Computer Conference, Vol. 33, (1968), 395--410.Google ScholarDigital Library
- Gaver, B., Dunne, T., and Pacenti, E. 1999. Design: Cultural probes. ACM Interactions 6:1, (1999), 21--29. Google ScholarDigital Library
- Gould, J.D. How to design usable systems. in R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, Morgan-Kaufmann, (1996), 93--121. Google ScholarDigital Library
- Greenberg, S. Teaching Human Computer Interaction to Programmers. 3(4), ACM Interactions, (1996), 62--76. Google ScholarDigital Library
- Greenberg, S. 2008. Embedding a design studio course in a conventional Computer Science program. P. Kotzé. W. Wong, J. Jorge, A. Dix and P. Silva (eds): Creativity and HCI: From Experience to Design in Education -- Selected Contributions from HCIEd 2007.Google Scholar
- Greenberg, S. and Thimbleby, H. The weak science of human-computer interaction. Proc CHI '92 Research Symposium on HCI (1992).Google Scholar
- Gutwin, C. and Greenberg, S. The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. Proc 9th IEEE Int'l Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE'00). (2000). Google ScholarDigital Library
- Hewett, Baecker, Card, Carey, Gasen, Mantei, Perlman, Strong and Verplank (1996) ACM SIGCHI Curricula for Human--Computer Interaction. Last updated 2004-06-03. http://sigchi.org/cdg/index.htmlGoogle Scholar
- Kaye, J. and Sengers, P. The Evolution of Evaluation. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/ Google ScholarDigital Library
- Landauer, T. The Trouble with Computers: Usefulness, Usability, and Productivity. Cambridge, MA: MIT Press. (1995) Google ScholarDigital Library
- Lieberman, H. The Tyranny of Evaluation. web.media.mit.edu/~lieber/Misc/TyrannyEvaluation.html, ACM CHI Fringe, (2003).Google Scholar
- Newman, W. CHI Guide to a Successful Archive Submission. http://www.chi2008.org/archiveGuide.html, (2008).Google Scholar
- Nielsen, J. Usability Engineering. Morgan Kaufmann. (1993). Google ScholarDigital Library
- Olsen Jr., D. (2007). Evaluating User Interface Systems Research. Proc ACM UIST'07. ACM Press. 251--258. Google ScholarDigital Library
- Pinelle, D. and Gutwin, C. A Review of Groupware Evaluations. Proc 9th IEEE Int'l Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE'00). (2000). 86--91. Google ScholarDigital Library
- Snodgrass, A. & Coyne, R. Interpretation in architecture: Design as a way of thinking. London: Routledge. (2006).Google Scholar
- Stanley Dicks, R. Mis-Usability: On the Uses and Misuses of Usability Testing. Proc ACM SIGDOC, (2002) Google ScholarDigital Library
- Sutherland, I. Sketchpad: A man-machine graphical communication system. PhD Thesis, MIT, (1963).Google ScholarDigital Library
- Suwa and Tverskey. External representations contribute to the dynamic construction of ideas. In M. Hegarty, B. Meyer, and N. H. Narayanan (Eds.), Diagrams NY: Springer-Verlag, (2002), 341--343. Google ScholarDigital Library
- Thimbleby, H. User Interface Design. ACM Press Frontier Series, Addison-Wesley, (1990). Google ScholarCross Ref
- Tohidi, M., Buxton, W., Baecker, R., and Sellen, A. User Sketches: A Quick, Inexpensive, and Effective way to Elicit More Reflective User Feedback. Proc. NordiCHI (2006), 105--114. Google ScholarDigital Library
- Tohidi, M., Buxton, W., Baecker, R. and Sellen, A. Getting the Right Design and the Design Right: Testing Many is Better than One. Proc ACM CHI, (2006), 1243--1252. Google ScholarDigital Library
- Weightman, G. (2003) Signor Marconi's Magic Box. Da Capo Press.Google Scholar
- Zhai, S. Evaluation is the worst form of HCI research except all those other forms that have been tried, essay published at CHI Place, (2003). http://www.almaden.ibm.com/u/zhai/publications.htmlGoogle Scholar
Index Terms
- Usability evaluation considered harmful (some of the time)
Recommendations
Dogmas in the assessment of usability evaluation methods
Usability evaluation methods (UEMs) are widely recognised as an essential part of systems development. Assessments of the performance of UEMs, however, have been criticised for low validity and limited reliability. The present study extends this ...
Iterative usability testing as continuous feedback: a control systems perspective
This paper argues that in the field of usability, debates about number of users, the use of statistics, etc. in the abstract are pointless and even counter-productive. We propose that the answers depend on the research questions and business objectives ...
Usability testing for oppression
This study examines a document produced by the United States Department of Homeland Security handed out to immigrant parents during the "Family Separation Policy" crisis of 2018. The article examines whether such a document could be ethically tested for ...
Comments