Short Scales of Satisfaction Assessment: A Proxy to Involve Disabled Users in the Usability Testing of Websites

Short scales of user satisfaction analysis are largely applied in usability studies as part of the measures to assess the interaction experience of users. Among the traditional tools, System Usability Scale (SUS), composed of 10 items, is the most applied quick evaluation scale. Recently, researchers have proposed two new and shorter scales: the Usability Metric for User Experience (UMUX), composed of four items, and the UMUX-LITE, which consists of only the two positive items of UMUX. Despite their recent creation, researchers in human-computer interaction (HCI) have already showed that these two tools are reliable and strongly correlated to each other [1‐3]. Nevertheless, there are still no studies about the use of these questionnaires with disabled users. As HCI experts claim [4‐7], when disabled and elderly users are included in the assessment cohorts, they add to the overall analysis alternative and extended perspectives about the usability of a system. This is particularly relevant to those interfaces that are designed to serve a large population of end-users, such as websites of public administration or public services. Hence, for a practitioner adding to the evaluation cohorts a group of disabled people may sensibly extend number and types of errors identified during the assessment. One of the major obstacles in creating mixed cohorts is due to the increase in time and costs of the evaluation. Often, the budget does not support the inclusion of disabled users in the test. In order to overcome these hindrances, the administering to disabled users of a short questionnaire—after a period of use (expert disabled costumers) or after an interaction test performed through a set of scenario-driven tasks (novice disabled users)—permits to achieve a good trade-off between a limited effort in terms of time and costs and the advantage of evaluating the user satisfaction of disabled people in the use of websites. To date, researchers have neither analyzed the use of SUS, UMUX, and UMUX-LITE by disabled users, nor the reliability of these tools, or the relationship among those scales when administrated to disabled people.

In this paper, we performed a usability test with 10 blind and 10 sighted users on the Italian website of public train transportation to observe the differences between the two evaluation cohorts in terms of: (i) number of identified errors, (ii) average score of the three questionnaires, and (iii) reliability and correlation of the three scales.

The outcomes confirmed that the three scales, when administered to blind or sighted users, are reliable (Cronbach’s α > 0.8), though UMUX reliability with disabled users is lower than expected (Cronbach’s α < 0.5). Moreover, all the scales are strongly correlated (p < .001) in line with previous studies. Nevertheless, significant differences were identified between sighed and blind participants in terms of (i) number of errors experienced during the interaction and (ii) average satisfaction rated through the three questionnaires. Our data show, in agreement with previous studies, that disabled users have divergent perspectives on satisfaction in the use of a website. The insight of disabled users could be a key factor to improve the usability of those interfaces which aim to serve a large population, such as websites of public administration and services. In sum, we argue that to preserve the budget and even incorporate disabled users’ perspectives in the evaluation reports with minimal costs, practitioners may reliably test the satisfaction by administrating SUS and UMUX or UMUX-LITE to a mixed sample of users with and without disability.