Using protocol analysis to evaluate the usability of a commercial web site
Introduction
Due to the popularity of the World Wide Web, many organizations are developing commercial web sites and adding web interfaces for their existing information systems. But very little research is concerned with the systematic evaluation of the usability of these web-based systems. Usability refers to how well and how easily a user, without formal training, can interact with an information system or a web site.
Most of the recent web site usability studies are focused on how users search for information on the web (e.g. [10], [13]). But very few of those studies deal with how users order products through web sites (for an exception see [21]). Clearly, better usability will result in more efficient interaction between the user and the site and will increase the probability that the user will return and/or make a purchase [12].
The aim of this paper is to apply the protocol analysis methodology to assess the usability of a commercial web site. To this end, we present the results of a study that systematically observed and analyzed a sample of users ordering free electronic greetings through a greeting card web site. Protocol analysis also known as the “think aloud” method is a qualitative research technique, well suited to determine how users interact with web sites, how users feel about a site (very important when use is discretionary), and how and when usability problems occur.
This paper is organized as follows. The next section reviews the literature on usability and protocol analysis. Section 3 justifies the selection of the product and the web site. Section 4 describes the methodology and the data analysis. Section 5 presents and discusses the results. The last two sections address the implications and the conclusions of this research.
Section snippets
Usability
It is well known that an information system should provide the necessary functions, so the users can accomplish their tasks. But functionality alone is not enough. The determinants of system acceptance are functionality (the degree in which the system provides the functions the users need to do their task) and usability (ease of use, a user-related and a task-related concept). Complete functionality cannot be achieved if the system is not usable [8]. Thus, “usability must be elevated to the
Product and web site selection
Tilson et al. [21] identify security concerns as the primary reason why more people refrain from purchasing items online. Therefore, a very convenient way to examine user interaction with commercial web sites is to select free Internet-based products, where users are not concerned about giving out their credit card information. Electronic greetings are one of such products. Another advantage of working with electronic products is that they are ordered and delivered online, overcoming many of
Research method
Before conducting the protocol analyses, we became thoroughly familiar with the site and the process of sending greetings. We also conducted a pilot study with undergraduate students totally unfamiliar with e-greetings’ web sites. Four sites were evaluated in the pilot (AmericanGreetings, BlueMountain, Egreetings Network and Hallmark). Each student was assigned to evaluate a site that was new to him or her. They were asked to use the site, state what they liked and disliked about it and if they
Results
Six of the eight subjects completed the assignment in an average time of 18 min. One subject took a very short time (only 5 min) because she had a hard time verbalizing despite the repeated reminders from the observer. Another subject took about 40 min because the session took place during a peak time on the web.
Implications
This research has theoretical and practical implications. At the theoretical level, the application of protocol analysis to evaluate the usability of a commercial web site represents a systematic attempt to collect data about the direct experience of the user and understand how objective usability factors affect perceived ease of use. In terms of the antecedents of ease of use of the TAM (Fig. 1), protocol analysis seems to be one of the best techniques to examine the interaction among the
Conclusions
In commercial web sites, whose use is discretionary and there is no possibility of user training, usability testing is of the utmost importance. Better usability will result in more successful interaction between the user and the site and may help the user to make a purchase. However, traditional usability tests can be difficult to conduct when the user population is widespread and diverse, as it is the case for commercial web sites.
This study has shown how think aloud protocol analyses can be
Acknowledgements
The author is grateful to Eliezer Fich, Tony Loviscek and David Rosenthal for their useful comments and suggestions and to the participants who made this research possible. Many thanks to the anonymous reviewers for their contributions to improve this manuscript.
Raquel Benbunan-Fich is an Assistant Professor of Information Systems at the Stillman School of Business at Seton Hall University. She received her PhD in Management Information Systems from Rutgers University in 1997. Her research interests include e-commerce, evaluation of Web-based systems, educational applications of computer-mediated communication systems and Asynchronous Learning Networks. She has published articles on related topics in Group Decision and Negotiations, Journal of Applied
References (25)
- et al.
Software tasks: intellectual or clerical
Information & Management
(1992) - P.S. Adler, T.A. Winograd, Usability: Turning Technologies into Tools, Oxford University Press, New York, NY,...
Electronic greetings versus paper cards: managerial challenges and entrepreneurial opportunities
Journal of Applied Management and Entrepreneurship
(1999)The use of protocol analysis in accounting
Accounting and Finance
(1985)- et al.
Exploring a word processor
Human-Computer Interaction
(1985) - J.S. Dumas, J.C. Redish, A Practical Guide to Usability Testing, Ablex Publishing Corporation, Norwood, NJ,...
- K.A. Ericsson, H.A. Simon, Protocol Analysis: Using Verbal Reports as Data, Second Printing, Oxford University Press,...
- N. Goodwin, Functionality and Usability, Communications of the ACM, March 1987, pp....
- et al.
A comparison of four prominent user-based methods for evaluating the usability of computer software
Ergonomics
(1995) - et al.
A heuristic evaluation of a World Wide Web prototype
Interactions Magazine
(1996)
Learning to use word processors: problems and prospects
ACM Transactions on Office Automation
Cited by (0)
Raquel Benbunan-Fich is an Assistant Professor of Information Systems at the Stillman School of Business at Seton Hall University. She received her PhD in Management Information Systems from Rutgers University in 1997. Her research interests include e-commerce, evaluation of Web-based systems, educational applications of computer-mediated communication systems and Asynchronous Learning Networks. She has published articles on related topics in Group Decision and Negotiations, Journal of Applied Management and Entrepreneurship, Journal of Computer Information Systems and Journal of Computer-Mediated Communication.