The effect of decision support system expertise on system use behavior and performance

doi:10.1016/j.im.2008.04.003

Information & Management

Volume 45, Issue 6, September 2008, Pages 349-358

https://doi.org/10.1016/j.im.2008.04.003 Get rights and content

Abstract

We performed an empirical investigation into the effect of users’ decision support system (DSS) expertise on their problem-solving strategies. The results indicated that individuals who had only recently learned to use the DSS were confused or restricted by the set of functions provided by the system and did not plan well for their use of the DSS. Those who had previous knowledge of the system exhibited more focused and efficient problem-solving behavior. Our findings suggested that problem-solving strategies depended significantly on the user's level of system expertise.

Introduction

We are still trying to understand how a decision support system (DSS) helps users improve their decision-making [16], [22]. Previous studies have reported that some DSS usage leads to performance improvement, while other usage resulted in little or no effect, or sometimes worse performance [1]. Studies have shown that DSS use does not always improve decision quality, but it can help users develop a better understanding of the decision problem [12], [23]. Thus, it is important to understand when and how a DSS helps users make better decisions [8]. We therefore explored the effect of the level of a person's expertise in the use of a DSS on consistent interpretation of its results.

The effect of expertise in the use of a DSS on decision-making strategy and resulting quality has been ignored: most laboratory experiments have tested a DSS only once (possibly with a few pre-tests), comparing a control group with a treatment group [11]. This lack of studies on the long-term use of DSS has been attributed to the cost of long-term investigations [9]. Ignoring system expertise can bias research results in two ways, there may be: (i) a lack of expertise among subjects; and (ii) high variance in their levels of expertise. Thus, if subjects are still at a novice level and cannot utilize the complete functionality of the DSS, the study may not be meaningful. Equally, if subjects have different levels of system knowledge, their joint effectiveness will be difficult to assess. Therefore, we believed that it was important to study DSS use when the user expertise was beyond a mere level of being “comfortable” with it.

Kohli and Devaraji [10] observed that the use of DSS over an extended period of time lead to improved decision outcomes. Our investigation went beyond this and tried to answer three questions:

•
Do subjects need a longer learning period than the “warm up” typically granted them?
•
Will users behave differently using DSS and solving problems when they are more experienced with the systems? And how are the differences represented (e.g., by choice of functions, demands on the interface, decision strategy, etc.)?
•
Does a higher level of system expertise lead to better decision-making?
•
We focused on the developing system expertise through repetitive experiments. Further, our study employed a control group to detect the effect of system expertise. Lastly, we examined the effect of system expertise, by collected data from system-generated logs of performance (such as time taken) instead of using subjective measures.

Section snippets

System restrictiveness

DSS restrictiveness is “the degree to which and the manner in which a DSS limits its users’ decision-making processes to a subset of all possible processes [20]” Decision-makers are thus obviously restricted to using only the functions designed into the system, and these restrictions influence the users’ decision processes. In empirical studies, system restrictiveness was found to be related to how individuals use the system [14]. Nakatsu and Benbasat [15] found that less restrictive systems

Research methodology

To test our hypotheses, we carried out an empirical study that compared the performance of two decision maker groups in a decision task during six sessions over the course of two weeks. One group was supported by a DSS throughout the experiment; the other group used the DSS only in the last session. The model underlying our research design is illustrated in Fig. 2.

System expertise development

It was necessary to check whether the system group developed system expertise during the first five sessions. Only if system learning occurred would it be meaningful to compare the measures in the sixth round in assessing differences between system experts and novices.

Our analysis demonstrated that, after the third session, subjects apparently had acquired system expertise, as indicated by a large drop in errors. However, one of the two measures of learning, simple errors, did not reflect this

Discussion and conclusions

Table 4 reviews the hypotheses. The table shows that system expertise led to different system use and problem-solving behaviors, as hypothesized; however, there was no difference in performance between experts and novices, suggesting that system knowledge did not translate into performance difference.

System novices seemed to be restricted by the set of functions provided by the system and showed less planning for problem solving. The system experts’ focus on a smaller number of functions (i.e.,

Acknowledgement

This research was supported by Seoul Future Contents Convergence Cluster established by Seoul &BD Program.

Zoonky Lee is a professor of Information Systems at Yonsei University in Korea. He holds a PhD degree from University of Southern California. His current research interests include IT's role in service innovation and participatory management in the context of web2.0. He has published in various journals including European Journal of Information systems, Information and Management, Journal of Organizational Computing and Electronic Commerce, Journal of Information Technology, Communications of

References (24)

J.H. Bradley
Analyzing the structure of expert knowledge
Information & Management
(2006)
C. Dowling et al.
Audit support systems and decision aids: current practice and opportunities for future research
International Journal of Accounting Information Systems
(2007)
G. Gunzelmann et al.
Problem solving: increased planning with practice
Cognitive Systems Research
(2003)
S.-Y. Hung
Expert versus novice use of the executive support systems: an empirical study
Information & Management
(2003)
M. Igbaria et al.
The impact and benefits of a DSS: the case of Fleet Manager
Information and Management
(1996)
R. Kohli et al.
Contribution of institutional DSS to organizational performance: evidence from a longitudinal study
Decision Support System
(2004)
D.H. Power et al.
Impacts of problem structure and computerized decision aids on decision attitudes and behaviors
Information and Management
(1994)
M.L. Williams et al.
The impact of DSS use and information load on errors and decision quality
European Journal of Operational Research
(2007)
I. Vessey
The effect of information presentation on decision making: a cost-benefit analysis
Information & Management
(1994)
B.R.N Benbasat
An evaluation of empirical research in managerial support systems
Decision Support Systems
(1990)

K. Elifson et al.

Fundamentals of Social Statistics

(1998)

T. Van Gog

Uncovering expertise-related differences in troubleshooting performance: combining eye movement and concurrent verbal protocol data

Applied Cognitive Psychology

(2005)

Cited by (22)

Business intelligence & analytics in management accounting research: Status and future focus
2018, International Journal of Accounting Information Systems
Citation Excerpt :
These studies mainly focused on what factors affect the successful use of BI&A in organizations – including the accounting function. According to the reviewed studies, users' level of expertise (Lee et al., 2008), satisfaction (Hou, 2012), and the technical problems they encounter (Deng and Chi, 2012) affect system use and the decision quality. Specifically, research shows that experts who are more efficient and effective in executing tasks compared to novice users deploy strategies that are more accurate and require less information processing (Hung, 2003).
Executives see technology, data and analytics as a transforming force in business. Many organizations are therefore implementing business intelligence & analytics (BI&A) technologies to support reporting and decision-making. Traditionally, management accounting is the primary support for decision-making and control in an organization. As such, it has clear links to and can benefit from applying BI&A technologies. This indicates an interesting research area for accounting and AIS researchers. However, a review of the literature in top accounting and information systems journals indicates that to date, little research has focused on this link. This article reviews the literature, points to several research gaps and proposes a framework for studying the relationship between BI&A and management accounting.
Automation bias: Empirical results assessing influencing factors
2014, International Journal of Medical Informatics
Citation Excerpt :
There was a slight but significant negative correlation between susceptibility to change and experience level of the physicians. Lee et al. [23] performed an empirical investigation into the effect of users’ DSS expertise on their problem-solving strategies. The results indicated that individuals who had only recently learned to use the DSS were confused or restricted by the set of functions provided by the system and did not use the DSS appropriately.
To investigate the rate of automation bias – the propensity of people to over rely on automated advice and the factors associated with it. Tested factors were attitudinal – trust and confidence, non-attitudinal – decision support experience and clinical experience, and environmental – task difficulty. The paradigm of simulated decision support advice within a prescribing context was used.
The study employed within participant before–after design, whereby 26 UK NHS General Practitioners were shown 20 hypothetical prescribing scenarios with prevalidated correct and incorrect answers – advice was incorrect in 6 scenarios. They were asked to prescribe for each case, followed by being shown simulated advice. Participants were then asked whether they wished to change their prescription, and the post-advice prescription was recorded.
Rate of overall decision switching was captured. Automation bias was measured by negative consultations – correct to incorrect prescription switching.
Participants changed prescriptions in 22.5% of scenarios. The pre-advice accuracy rate of the clinicians was 50.38%, which improved to 58.27% post-advice. The CDSS improved the decision accuracy in 13.1% of prescribing cases. The rate of automation bias, as measured by decision switches from correct pre-advice, to incorrect post-advice was 5.2% of all cases – a net improvement of 8%.
More immediate factors such as trust in the specific CDSS, decision confidence, and task difficulty influenced rate of decision switching. Lower clinical experience was associated with more decision switching.
Age, DSS experience and trust in CDSS generally were not significantly associated with decision switching.
This study adds to the literature surrounding automation bias in terms of its potential frequency and influencing factors.
A comparison of representations for discrete multi-criteria decision problems
2013, Decision Support Systems
Citation Excerpt :
Heatmaps exhibit both characteristics by enabling the visualization of high density information and providing exact data values in the cells. Research has shown that expertise with the support provided leads to a reduction in decision time [14,34,43]. As we expect DMs to be familiar with tables and PCP but not with heatmaps, the use of heatmaps should result in longer decision time.
Discrete multi-criteria decision problems with numerous Pareto-efficient solution candidates place a significant cognitive burden on the decision maker. An interactive, aspiration-based search process that iteratively progresses toward the most preferred solution can alleviate this task. In this paper, we study three ways of representing such problems in a DSS, and compare them in a laboratory experiment using subjective and objective measures of the decision process as well as solution quality and problem understanding. In addition to an immediate user evaluation, we performed a re-evaluation several weeks later. Furthermore, we consider several levels of problem complexity and user characteristics. Results indicate that different problem representations have a considerable influence on search behavior, although long-term consistency appears to remain unaffected. We also found interesting discrepancies between subjective evaluations and objective measures. Conclusions from our experiments can help designers of DSS for large multi-criteria decision problems to fit problem representations to the goals of their system and the specific task at hand.
The role of post-use trust in the acceptance of a technology: Drivers and consequences
2011, Technovation
The aim of this paper is to shed light on the role of trust in the acceptance of a technology once the firm has already used it (post-use trust). First, we examine the drivers which encourage the firm to trust in a technology and, second, we study the consequences of trust in terms of firm performance and intentions to continue to use the technology. The drivers involve (1) perceptions experienced by the firm during the employment of the technology, (2) the influence of the firm's trading partners and (3) the importance of the IT manager's attitude towards the technology. The study focuses on the acceptance of e-invoicing in Spain and the sample consists of 100 early adopter firms who have perceptions and knowledge about this technology. Our findings clarify the relationship between trust and technology acceptance and have important implications about trust-building structures that could improve the application of technologies that are in the early stages of implementation.
Online government advisory service innovation through Intelligent Support Systems
2011, Information and Management
Citation Excerpt :
In the literature, there have been confounding results reported on the impact of ISS use on performance. Some researchers have found no significant improvements in decision quality through the use of ISS, while others saw an increase in decision quality by users [14]. Our study therefore examined this issue by trying to determine what kinds of ISS would lead to better performance.
Motivated by lessons learnt from countries with a high e-government readiness index, we used the lens of ‘empowerment’ to understand the ways in which online Intelligent Support Systems can be successfully developed. We found that decision transparency and decision making satisfaction played key roles providing a higher sense of empowerment to the public, including their having a greater perceived power relationship with the government agency and a better feeling of being in control. This approach has improved citizens’ positive perceptions of the government agencies as service providers. These findings make a strong case for the investment in the ISS and have provided useful information for government agencies who wish to develop or improve their online advisory services.
Situated DSS for personal finance management: Design and evaluation
2010, Information and Management
The Situated Decision Support System (SDSS) model is a type of DSS that maintains close links with the target environment and has capabilities for sensing, monitoring, decision support, and limited decision-making, action generation, and implementation. But though a generic description of SDSS had been provided, no empirical test has been made to prove its value. We performed experiments using human subjects to test an SDSS prototype. Personal finance management was chosen as an application domain, and the design and implementation of the SDSS prototype is discussed. The experiments involve subjects who carry out their normal shopping tasks in a simulated setup. One group of subjects was provided with SDSS support, while the others used a traditional decision support model. Overall, the results attest to the superiority of the SDSS model in terms of key decision performance variables.

View all citing articles on Scopus

Christian Wagner is a professor in City University of Hong Kong's Information Systems Department. He also teaches in the University of Toronto's Global EMBA (Omnium GEMBA) program, in which City University is a partner institution. Chris presently leads a consultancy project on knowledge management for the Society for Information Management's Advanced Practices Council. Previously he carried out consulting projects with clients such as The Sabre Group, Motorola, Deloitte and Touche, and Daimler-Benz (now Daimler-Chrysler). Chris has worked on several knowledge management and business intelligence projects, and has authored or co-authored more than 60 articles in journals, conferences, and books. He also serves as an editorial board member for several academic journals.

Ho Kyoung Shin is currently a research professor with the graduation school of Information at Yonsei University. She received her PhD in MIS from Yonsei University in Seoul. Her research interests include knowledge management, virtual world, and IT outsourcing. Her work appeared in various journals including the Journal of MIS Research, Small Business Research, and the Journal of CALS/EC.

View full text

The effect of decision support system expertise on system use behavior and performance

Abstract

Introduction

Section snippets

System restrictiveness

Research methodology

System expertise development

Discussion and conclusions

Acknowledgement

Information & Management

International Journal of Accounting Information Systems

Cognitive Systems Research

Information & Management

Information and Management

Decision Support System

Information and Management

European Journal of Operational Research

Information & Management

An evaluation of empirical research in managerial support systems

Decision Support Systems

Fundamentals of Social Statistics

Uncovering expertise-related differences in troubleshooting performance: combining eye movement and concurrent verbal protocol data

Applied Cognitive Psychology