Introduction

Information Systems are increasingly employed in healthcare industry. Therefore, it is necessary for the designers of information technology and communication systems to assess them continuously [1]. However, it is impossible to evaluate such systems without first analyzing their users’ understanding of them. Moreover, users’ satisfaction guarantees the successful implementation of information systems [2].

An information system is a combination of technology, individuals and processes used to transport, store, manipulate and present data [3]. Health information refers to the organized data related to a patient or a group of patients [4].

Health information systems were presented in early 1970’s [5]. Healthcare information systems include data and concepts in health services given to patients to improve the management of such services [6]. Hospital information systems have many functions such as patients’ admission, transfer and discharge as well as financial and clinical affairs [6–8]. There are many advantages to health information services: the enhancement of medical care quality and reduction of costs and errors. Most evaluations performed in this field have focused on the reduction of costs and the improvement of quality [9]. Certain studies, however, have emphasized the technical and social aspects [10].

Hospital information systems (HIS) have many advantages: they increase the quality of medical care and reduce costs and clinical errors [11]. However, despite all these benefits, information systems have also some problems [12]. To recognize and remove such problems, it is necessary to evaluate HIS permanently. In fact, evaluation in an endless process [13] and its results can be used to make decisions regarding the required investments for designing hospital information systems [14].

It is necessary to consider the evaluation of hospital information systems before, during and after developing them [15]. Financial aspects and patient satisfaction are also important factors in the evaluation of HIS [16].

There are many articles focusing on the evaluation of health information systems, as well as their indicators and outcomes [17]. Littlejohns et al. (2003) evaluated the integrated computer system in South Africa [18]. In their evaluation, they focused on optimal education, change management and support, project management, improvement of the link systems, maintenance of information confidentiality, standardization of processes to manage patients, increase of revenue and cost of each service package.

UK Health Informatics Institute presents a framework for evaluating health information systems. In this framework, questions, tools, techniques and manner of evaluation are provided [19]. A system is considered weak if it does not provide its users with proper information. For the information system to be successful, not only the users’ information needs should be met, but the manner of interaction between the system and its users should also be considered.

A technology is accepted by users if it is usable and helpful [11]. In other words, the quality of a system depends on its usability, availability, reliability, adaptability and response time [20]. The purpose of usability evaluation is to determine the strong and weak points of information systems and to provide guidelines for the improvement of their application [21]. Ergonomic quality of software systems should be evaluated with reference to their application context (including user, task, equipment and environment), user type and users’ computer experience.

ISO 9241 presents the ergonomic requirements for designing software. ISO 9241 has 17 parts and seven principles. The 17 parts consist of general introduction, general guidance on task requirements, visual display requirements, keyboard requirements, workstation layout and postural requirements, environmental requirements, display requirements with reflection, requirements for displayed colors, requirements for non-keyboard input devices, dialog principals, usability statements, presentation of information, user guidance, menu dialogs, command dialogs, direct manipulation dialogs, and form filing dialogs. And the seven principles are suitability for the task, self-descriptiveness, controllability, conformity with user expectations, error tolerance, suitability for individualizations, and suitability for learning. These seven principles are considered for usability evaluation.

IsoMetrics is a reliable and suitable technique for HIS evaluation [15, 21]. It is a technique to evaluate the usability of software applications with respect to the international standard ISO 9241 part 10. IsoMetrics is user-oriented. Due to the two versions of IsoMetrics evaluation—formative and summative-, two kinds of IsoMetrics—short and long—are presented, [21] Which apply to summative and formative evaluations, respectively. Summative assessment is conducted some years after the implementation of information system [14]. Formative or constructive assessment is more qualitative. It is conducted during the engineering life cycle of the information system or before its further development [21].

In recent years, many researchers have evaluated usability of clinical information systems [22–26]. The results of the studies conducted on the usability evaluation of health information systems show that little attention has been paid to usability [27–30].

IS-H*med, SAP-HR and Microsoft Word for Windows (Version 2.0) are the three softwares which were evaluated using IsoMetrics in 1999, 2000, and 2004, respectively [15, 21]. The year of evaluation, name, number of users, and application of each software are presented in Table 1.

Table 1 Software’s evaluated by IsoMetrics
Full size table

In this article, the evaluation results of the three softwares—IS-H*med, SAP-HR and Microsoft Word (Version 2.0)—are compared with that of Iran’s HIS.

Materials and methods

In this research used a usability questionnaire based on ISO 9241 part 10 so that the usability of hospital information systems could be investigated. The questionnaire consisted of 75 items based on the seven principles of suitability for the task, self-descriptiveness, controllability, conformity with user expectations, error tolerance, suitability for individualization, and suitability for learning. These principles are defined briefly in Table 2 [21].

Table 2 The seven principles of ISO 9241 part 10
Full size table

It is to be mentioned that the 75-item questionnaire included 15 questions for suitability for the task, 12 questions for self-descriptiveness, 11 questions for controllability, 8 questions for conformity with user expectations, 15 questions for error tolerance, 6 questions for suitability for individualization, and 8 questions for suitability for learning.

Each question of the IsoMetrics Questionnaire is assessed on a five-point rating scale starting from 1 (“predominantly disagree”) to 5 (“predominantly agree”). The data were analyzed using SPSS and Excel softwares. Questionnaires with more than 20% of missing data (more than 15 unanswered items) were not analyzed [15]. In case of less than or equal to 15 omissions, the missing values were replaced by the mean scale value (3) of the items. Some items of the questionnaire (A1, A8, T12, E8, F1, F7, F14, L1, and L7) were negative in nature. The values of these items were inverted by the transformation ri’ = 6−ri for further analysis [21]. For statistical analyses, one-sample t-test and p-values were used.

This research was conducted in two stages. First, the hospital information system (HIS) of Iran was evaluated. Then, the results of the first stage were compared with those of the three softwares which were evaluated by the same IsoMetrics. A multi-step sampling was conducted for the first stage of the study; that is, first the capital’s hospitals having computerized hospital information systems were determined. Then a list of companies providing such systems was prepared. And finally, for each company, one hospital was selected.

These companies included Rayavaran, Tarrahane Boo Ali, Rahavard Rayane, Data Link Informatics, Teb-o Rayane, Tirajhe Rayane, Pouya Padideh, Kian Tak, Smart Chip. And the hospitals consisted of Firoozgar, Toos, Shariati, Rasoul, Hashemi Nejad, Children Medical Center, Kasra, Vali-e Asr, Kashani, Day, Hazrat-e Fatima, Atieh, and Imam Reza.

To conduct the second stage of the study, the HIS IsoMetrics mean value were compared with those of the three other applications, i.e., IS-H*med, SAP-HR, and Microsoft word. IS-H*med, SAP-HR, and Microsoft word were the softwares evaluated using ISO 9241 part 10 by Hamborg et al. in 2004, Gruber in 2000, and Gediga et al. in 1999, respectively.

The viewpoints of three types of users (nurses, users of paraclinic units, and department secretaries) were studied using a standardized IsoMetrics questionnaire. For that purpose, one nurse was selected for every ten beds and one user from every paraclinic unit. However, the viewpoints of all department secretaries were studied. In total, 285 individuals including 98 nurses, 88 department secretaries and 77 users of paraclinic units filled out the standardized IsoMetrics questionnaire. There were 22 unidentified user types. In Table 3, the frequency and percentage of the three user types are presented. It is to be mentioned that 22 of the questionnaires were filled out by unidentified user types.

Table 3 Frequency and percentage of the user types
Full size table

More than half of the users (56.8%) were nurses. Also, most users (62.5%) had worked for 1–9 years and 57 users (31%) for 10–19 years in their current positions. In addition, most users (274 individuals, i.e., 88.2%) were female. Moreover, the subjects were mostly (119 individuals, i.e., 43.6%) in the age group of 30–39 years old. Furthermore, most of them (65.5%) had a BS degree. Regarding computer literacy, some 66% of users did not have enough computer literacy to use information systems. However, 178 users (62.7%) declared that their computer literacy was at the average level. Among them, only 90 users (34.1%) had the ICDL (International Computer Driving License) certificate.

Results

In this study, the viewpoints of three types of users about usability of hospital information systems were considered. Figure 1 shows the users’ viewpoints regarding the usability of HIS in Iran.

Fig. 1
figure 1

HIS IsoMetrics scale means

Full size image

The mean value of HIS was compared with that of IS-H*med, SAP-HR, and Microsoft word separately, using p-values and t-tests. Figure 1 shows that the value of scale “conformity with user expectations” is higher than the others; yet, compared with IS-H*med, it has a lower value. The comparison between HIS and IS-H*med reveals that HIS is significantly more usable than IS-H*med with respect to all principles except conformity with user expectations. Regarding this principle, the following are among the points to be considered consistently by HIS designers: predictability of the time needed to perform a task, anticipation of the screen in the following processing sequence, defining the same function keys throughout the program for the same functions, emergence of messages in the same location.

As mentioned earlier, IS-H*med, SAP-HR, and Microsoft Word for Windows, Version 2 were evaluated by Hamborg et al. [21], Gruber [31], and Gediga et al. [15], respectively. In Tables 4, 5, and 6, the IsoMetrics mean value of HIS (evaluated by the present researchers) is compared with that of the other three evaluations in terms of t, df (Degree of Freedom), p value and d.

Table 4 Comparison between HIS and IS-H*med
Full size table
Table 5 Comparison between HIS and SAP-HR
Full size table
Table 6 Comparison between HIS and Microsoft Word (Version 2.0)
Full size table

According to Table 5, HIS is significantly more usable on scales of suitability for the task, error tolerance, and suitability for learning.

As it can be seen in Table 6, usability of HIS is rated low from the users’ point of view. In fact, HIS does not meet the ergonomic quality of a standard windows application.

Figure 2 shows the IsoMetrics scale means of HIS, IS-H*med, SAP-HR and Microsoft Word (Version 2.0). Figure 2 and Tables 4 and 5 show that the usability of HIS is rated higher than that of IS-H*med and SAP-HR, from the users’ point of view. However, the word processing software is more usable than all these. In other words, from the users’ point of view, Microsoft Word meets the ergonomic quality more.

Fig. 2
figure 2

IsoMetrics scale means of HIS, and IS-H*med, SAP-HR and Microsoft Word (Version 2.0)

Full size image

Conclusions

IsoMetrics is a reliable and suitable technique for HIS evaluation [15, 21]. It is a technique to evaluate the usability of software applications with respect to the international standard ISO 9241 part 10. In this research compared results of evaluation of four softwares using IsoMetrics. The comparison between HIS and IS-H*med showed that the former was significantly rated higher than the latter on all IsoMetrics scales except “conformity with user expectations”. Therefore, it is suggested that hospital information system designers pay more attention to the following points:

  • Predictability of the time needed to perform a task

  • Anticipation of the screen in the following processing sequence

  • Defining the same function keys throughout the program for the same functions

  • Emergence of messages in the same location

The scale mean of suitability for the task in HIS is 3.04. It seems that HIS designers do pay attention to the suitability of HIS for the task, but apparently this much attention is not enough. Many research results have shown that inattention to the scale “suitability for the task” increases the amount of error, which will in turn have a negative impact on tasks [22, 24, 32].

Regarding the principle “suitability for the task”, one of the questions asked whether the software helped the users to perform their task. In response to the above question, 23.9% of the subjects of the present study believed that the HIS did not help them with performing their task; rather, they believed that it acted as an obstacle. Moreover, in a study as “A usability evaluation of four commercial dental computer-based patient record systems” conducted by Thyvalikakath et al. (2008), it was shown that 28% of users failed to perform their duties when using the system [27]. In their study, 30% of users even did their duties wrongly due to the information system. They pointed out that it was necessary to do more studies about documentation errors while using the hospital information system. Many studies also revealed that a decrease in the HIS usability led to an increase in the errors committed in performing the task [22, 24, 33, 34]. Such errors have a negative effect on the task output.

The mean of “suitability for learning” in HIS was rather low (2.98). However, in a study done by Thyvalikakath et al. on dental computer-based patient record system, suitability for learning was found to be satisfactory [27]. Therefore, it is suggested that the country’s HIS designers pay more attention to “suitability for learning”.

The results show that hospital information system in Iran is average in terms ergonomic quality. It is to be mentioned that the ergonomic quality of an information system should be determined with regard to its context [35]. That is why, in the present study, this quality was evaluated with regard to the user types (nurses, users of paraclinic units, and department secretaries).

This study was done based on IsoMetrics short which has certain weaknesses; i.e., the cause of users’ dissatisfaction and of the problems existing in the information system cannot be determined. To discover the cause of such problems and weak points, IsoMetrics long [21] is needed. In addition to this type of IsoMetrics, such techniques as user tests [36] and walkthroughs [33] can also be used.

Walkthrough is the superficial examination of a computer program before running it by reading the source code (program instructions). Walkthrough aims at identifying the possible errors in the initial stages of information system development. When we had received the evaluator reports we began the user tests, hoping to detect more possible problems in the use of terminology databases and at the same time compare the results the expert evaluation with the observations of the users [37].

One of the important things is that the evaluators must pay attention to the number of users. Cohan (1988) found out that the number of users was a factor affecting the results [38]. In this research study, the viewpoints of 285 users were considered while Gruber [31] and Gediga [15] studied the viewpoints of only 28 and 55 users in their studies, respectively.

It should be noted that the evaluation of information systems is a complex process because all human, technical and organizational aspects should be considered [39].