Elsevier

Information & Management

Volume 39, Issue 2, December 2001, Pages 151-163
Information & Management

Using protocol analysis to evaluate the usability of a commercial web site

https://doi.org/10.1016/S0378-7206(01)00085-4Get rights and content

Abstract

Despite the increasing popularity of electronic commerce, there appears to be little evidence of the methodical evaluation of the usability of commercial web sites. The usability of a web site defines how well and how easily a visitor, without formal training, can interact with the site. This paper reports the results of a research project, which applies a systematic qualitative technique known as protocol analysis or think aloud method, to examine the usability of a commercial web site. About 15 usability principles and 3 evaluation parameters (content, navigation and interactivity) were used as a framework to analyze the verbal protocols of a sample of users interacting with a greeting card web site. The protocols provided evidence of usability problems caused by crowded content, poor navigation and cumbersome interactivity. These results underscore the importance of two crucial usability goals for commercial web sites: clear path to products and transparency of the ordering process.

Introduction

Due to the popularity of the World Wide Web, many organizations are developing commercial web sites and adding web interfaces for their existing information systems. But very little research is concerned with the systematic evaluation of the usability of these web-based systems. Usability refers to how well and how easily a user, without formal training, can interact with an information system or a web site.

Most of the recent web site usability studies are focused on how users search for information on the web (e.g. [10], [13]). But very few of those studies deal with how users order products through web sites (for an exception see [21]). Clearly, better usability will result in more efficient interaction between the user and the site and will increase the probability that the user will return and/or make a purchase [12].

The aim of this paper is to apply the protocol analysis methodology to assess the usability of a commercial web site. To this end, we present the results of a study that systematically observed and analyzed a sample of users ordering free electronic greetings through a greeting card web site. Protocol analysis also known as the “think aloud” method is a qualitative research technique, well suited to determine how users interact with web sites, how users feel about a site (very important when use is discretionary), and how and when usability problems occur.

This paper is organized as follows. The next section reviews the literature on usability and protocol analysis. Section 3 justifies the selection of the product and the web site. Section 4 describes the methodology and the data analysis. Section 5 presents and discusses the results. The last two sections address the implications and the conclusions of this research.

Section snippets

Usability

It is well known that an information system should provide the necessary functions, so the users can accomplish their tasks. But functionality alone is not enough. The determinants of system acceptance are functionality (the degree in which the system provides the functions the users need to do their task) and usability (ease of use, a user-related and a task-related concept). Complete functionality cannot be achieved if the system is not usable [8]. Thus, “usability must be elevated to the

Product and web site selection

Tilson et al. [21] identify security concerns as the primary reason why more people refrain from purchasing items online. Therefore, a very convenient way to examine user interaction with commercial web sites is to select free Internet-based products, where users are not concerned about giving out their credit card information. Electronic greetings are one of such products. Another advantage of working with electronic products is that they are ordered and delivered online, overcoming many of

Research method

Before conducting the protocol analyses, we became thoroughly familiar with the site and the process of sending greetings. We also conducted a pilot study with undergraduate students totally unfamiliar with e-greetings’ web sites. Four sites were evaluated in the pilot (AmericanGreetings, BlueMountain, Egreetings Network and Hallmark). Each student was assigned to evaluate a site that was new to him or her. They were asked to use the site, state what they liked and disliked about it and if they

Results

Six of the eight subjects completed the assignment in an average time of 18 min. One subject took a very short time (only 5 min) because she had a hard time verbalizing despite the repeated reminders from the observer. Another subject took about 40 min because the session took place during a peak time on the web.

Implications

This research has theoretical and practical implications. At the theoretical level, the application of protocol analysis to evaluate the usability of a commercial web site represents a systematic attempt to collect data about the direct experience of the user and understand how objective usability factors affect perceived ease of use. In terms of the antecedents of ease of use of the TAM (Fig. 1), protocol analysis seems to be one of the best techniques to examine the interaction among the

Conclusions

In commercial web sites, whose use is discretionary and there is no possibility of user training, usability testing is of the utmost importance. Better usability will result in more successful interaction between the user and the site and may help the user to make a purchase. However, traditional usability tests can be difficult to conduct when the user population is widespread and diverse, as it is the case for commercial web sites.

This study has shown how think aloud protocol analyses can be

Acknowledgements

The author is grateful to Eliezer Fich, Tony Loviscek and David Rosenthal for their useful comments and suggestions and to the participants who made this research possible. Many thanks to the anonymous reviewers for their contributions to improve this manuscript.

Raquel Benbunan-Fich is an Assistant Professor of Information Systems at the Stillman School of Business at Seton Hall University. She received her PhD in Management Information Systems from Rutgers University in 1997. Her research interests include e-commerce, evaluation of Web-based systems, educational applications of computer-mediated communication systems and Asynchronous Learning Networks. She has published articles on related topics in Group Decision and Negotiations, Journal of Applied

References (25)

  • R.L. Glass et al.

    Software tasks: intellectual or clerical

    Information & Management

    (1992)
  • P.S. Adler, T.A. Winograd, Usability: Turning Technologies into Tools, Oxford University Press, New York, NY,...
  • R. Benbunan-Fich

    Electronic greetings versus paper cards: managerial challenges and entrepreneurial opportunities

    Journal of Applied Management and Entrepreneurship

    (1999)
  • M.J. Bouwman

    The use of protocol analysis in accounting

    Accounting and Finance

    (1985)
  • J.M. Carroll et al.

    Exploring a word processor

    Human-Computer Interaction

    (1985)
  • J.S. Dumas, J.C. Redish, A Practical Guide to Usability Testing, Ablex Publishing Corporation, Norwood, NJ,...
  • K.A. Ericsson, H.A. Simon, Protocol Analysis: Using Verbal Reports as Data, Second Printing, Oxford University Press,...
  • N. Goodwin, Functionality and Usability, Communications of the ACM, March 1987, pp....
  • R.D. Henderson et al.

    A comparison of four prominent user-based methods for evaluating the usability of computer software

    Ergonomics

    (1995)
  • M.D. Levi et al.

    A heuristic evaluation of a World Wide Web prototype

    Interactions Magazine

    (1996)
  • R. Mack et al.

    Learning to use word processors: problems and prospects

    ACM Transactions on Office Automation

    (1983)
  • New York Times, On-line merchants find that a well-designed web site can have a big impact on the bottom line,...
  • Cited by (0)

    Raquel Benbunan-Fich is an Assistant Professor of Information Systems at the Stillman School of Business at Seton Hall University. She received her PhD in Management Information Systems from Rutgers University in 1997. Her research interests include e-commerce, evaluation of Web-based systems, educational applications of computer-mediated communication systems and Asynchronous Learning Networks. She has published articles on related topics in Group Decision and Negotiations, Journal of Applied Management and Entrepreneurship, Journal of Computer Information Systems and Journal of Computer-Mediated Communication.

    View full text