Validating the external quality subcharacteristics of software products according to ISO/IEC 9126
Introduction
The competitive environment in the software market and the needs of customers require that software developers focus on user satisfaction as a measure of software product quality. User satisfaction is often considered as the ultimate aim of quality management and has a positive impact on organizational cost, profit, and sales growth [27]. However, users are often dissatisfied with software quality [20]. In fact, widespread dissatisfaction in the United States led to recent legislation governing software quality and suppliers' responsibility [21]. Thus, understanding the key quality attributes that influence user satisfaction can lead to a competitive advantage in the software market place by aiding management decisions such as resource allocation, aimed at improving software product quality [22].
Investigation of the quality attributes that influence user satisfaction requires the use of a quality model. A quality model normally defines a set of quality attributes that may be present in software product. In particular, because users place different values on each attribute depending on the product's use, it is important that quality attributes are clear to the consumer. Software engineering researchers and practitioners have suggested many different quality models each with a varying number of attributes [2], [11], [29]. Recently, in order to address the issues of software product quality in a consistent way, ISO/IEC 9126 (Software Product Quality) [17] defined six characteristics, 27 subcharacteristics, and their associated measures of software product quality. A summarized explanation of quality models and their attributes can be found in [23].
The proposition of ISO/IEC 9126 is that higher external quality (quality when software is executed) indicates higher ‘quality in use’. Subcharacteristics of external quality are of little value by themselves unless there is empirical evidence that they are associated with important characteristics such as user satisfaction, defined in ‘quality in use’.
The purpose of this study is to test the proposition by showing that each of the subcharacteristics of external quality is positively associated with user satisfaction, defined in ‘quality in use’. A univariate ordered probit regression [37] is applied to 75 user survey data based on the use of a query and reporting tool. Ordered probit regression is useful for analyzing datasets with a dependent variable of an ordered response scale of user satisfaction [15]. In addition, this study investigates the external quality subcharacteristics that strongly influence user satisfaction by using a multivariate ordered probit regression. Results can aid management decisions, such as resource allocation, aimed at improving software product quality.
ISO/IEC 9126 is a generic quality model in that the standard assumes its applicability to any software product by allowing it to be tailored for a specific purpose. Thus, the ISO/IEC 9126 standard has also been used in a number of previous studies: measuring the quality of architectural design [26], evaluating a digital video recorder system during its development stage [8], as an evaluation framework for more than 30 cases [36], and R&D project evaluation [30]. ISO/IEC 9126 also has been used for identifying relevant criteria for evaluating software products [32] and for selecting packaged software systems [14], [19]. Xenos and Christodoulakis [38] present a method to measure customer's perception of software quality.
In a survey of intranet applications, Leung [24] used an extended ISO/IEC 9126 model and identified five subcharacteristics (availability, accuracy, security, suitability, and time behavior) as the key attributes of intranet applications, where availability is an extended attribute of reliability. Stalhane et al. [34] present results of the PROFF project that the quality of service and follow-up activities are more important than the product quality. Kekre et al. [22] conducted an ordered probit regression to identify the importance of seven key satisfaction factors. Although the authors did not mention the ISO/IEC 9126 characteristics in their study, their model includes the ISO/IEC 9126 characteristics and an additional characteristic “quality of documentation”, where satisfaction for each characteristics is measured on a five-point rating scale.
This paper is organized as follows: Section 2 presents the research design of the study. Section 3 addresses data collection, measurement scale, reliability of measures, and imputation for tackling missing values. In Section 4, we present a brief explanation of an ordered probit regression model that we used in the study. Section 5 presents results. Section 6 discusses some of the issues and describes some limitations of the study. Lastly, Section 7 contains concluding remarks.
Section snippets
Theoretical basis
In an effort to identify attributes of software product quality that can be useful to developers, acquirers, and evaluators, a set of International Standards have been developed. They are ISO/IEC 9126: Part 1 (Quality model, 2001), Part 2 (External metrics, 2003), Part 3 (Internal metrics, 2003), and Part 4 (Quality in use metrics, 2004). ISO/IEC 9126-1 defines a quality model that includes six characteristics (FUNCTIONALITY, RELIABILITY, USABILITY, EFFICIENCY, MAINTAINABILITY, and
Data collection
A marketing department identified 200 users of its company's packaged software product — a query and reporting tool for business databases. The marketing department keeps a list of users for technical support purposes. Of the 200 people we contacted through email or telephone, 75 (38%) responded to the questionnaire. The respondents included 48 end-users, 25 developers who had modified the tool to embed it in their own applications, and two managers — all of whom we refer to as users.
Before
Analysis method
This study conducted an empirical validation of the external quality subcharacteristics in two stages [13]. The first stage was to examine the correlation coefficients between pairs of variables [31]. In a large enough sample size, even a very small association can be statistically significant [9]. In our study, we considered 0.3 as the minimal magnitude according to Cohen's guideline for interpreting the magnitude of the correlation coefficient [9]. Of course many other factors are expected to
Descriptive statistics
In order to capture the differential effect of the subcharacteristics of user response, demographic information was also collected including gender, business sector, training, year of usage, and respondent's job type (end-user, developer, and manager).
As shown in Table 1, the gender distribution of the respondents was 78.7% male and 21.3% female. In the classification of business sectors, 29 (38.7%) respondents were from Information Technology (IT) followed by manufacturing (26.7%), banking
Discussion and limitations
Security is a key aspect for many software products, e.g., network centric applications. However, neither the ordered probit regression study nor the correlation coefficient study suggest that security is related to user satisfaction. The software tool that we investigated has security aspects such as file encryption, access control, and others. We conducted a follow-up examination to explain the reason why security is not seen as an aspect of user satisfaction. First, we contacted several
Concluding remarks
No single study can be fully definitive and widely generalizable. Additional empirical work with larger and broader samples is necessary to help us better understand the relationship between subcharacteristics and USER SATISFACTION. Additional empirical work is also necessary to help us better understand the relationship in order to see if the results are consistent across different types of software product.
As future studies, the relationship between internal quality (non-executable software
Acknowledgement
The author wishes to thank Robin Hunter (formerly of the University of Strathclyde) for his valuable comments on an earlier version of this manuscript.
References (38)
- et al.
Exploring the relationship between design measures and software quality in objective-oriented systems
Journal of Systems and Software
(2000) Quality metrics for intranet applications
Information and Management
(2001)- et al.
ISO Quality Standards for Measuring Architecture
(2004) - et al.
AHP-based evaluation of AHP-Software
European Journal of Operational Research
(1999) - et al.
In search of the customer's quality view
Journal of Systems and Software
(1997) - et al.
Practical concurrent software evaluation for certification
Journal of Systems and Software
(1997) Survival Analysis Using the SAS System: A Practical Guide
(1995)Identifying quality-requirement conflicts
IEEE Software
(1996)- et al.
Maintainability: A Key to Effective Serviceability and Maintenance Management
(1995) - et al.
An Introduction to S-Plus and the Hmisc and Design Libraries