Skip to main content
SpringerLink
Log in

Inconsistency indices for pairwise comparison matrices: a numerical study

  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Evaluating the level of inconsistency of pairwise comparisons is often a crucial step in multi criteria decision analysis. Several inconsistency indices have been proposed in the literature to estimate the deviation of expert’s judgments from a situation of full consistency. This paper surveys and analyzes ten indices from the numerical point of view. Specifically, we investigate degrees of agreement between them to check how similar they are. Results show a wide range of behaviors, ranging from very strong to very weak degrees of agreement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Aguaròn, J., & Moreno-Jimènez, J. M. (2003). The geometric consistency index: approximated threshold. European Journal of Operational Research, 147(1), 137–145.

    Article  Google Scholar 

  • Anholcer, V., Babiy, V., Bózoki, S., & Koczkodaj, W. W. (2011). A simplified implementation of the least squares solution for pairwise comparisons matrices. Central European Journal of Operations Research, 19(4), 439–444.

    Article  Google Scholar 

  • Barzilai, J. (1998). Consistency measures for pairwise comparison matrices. Journal of Multi-Criteria Decision Analysis, 7(3), 123–132.

    Article  Google Scholar 

  • Bózoki, S. (2008). Solution of the least squares method problem of pairwise comparison matrices. Central European Journal of Operations Research, 16(4), 345–358.

    Article  Google Scholar 

  • Brunelli, M. (2011). A note on the article “Inconsistency of pair-wise comparison matrix with fuzzy elements based on geometric mean” [Fuzzy Sets and Systems 161 (2010) 1604–1613]. Fuzzy Sets and Systems, 161(1), 1604–1613. 2010, Fuzzy Sets and Systems, 176, 76–78.

    Google Scholar 

  • Brunelli, M., Critch, A., & Fedrizzi, M. (2013). On the proportionality between some consistency indices in the AHP. Applied Mathematics and Computation. doi:10.1016/j.amc.2013.01.036.

    Google Scholar 

  • Cavallo, B., & D’Apuzzo, L. (2009). A general unified framework for pairwise comparison matrices in multicriterial methods. International Journal of Intelligent Systems, 24(4), 377–398.

    Article  Google Scholar 

  • Cavallo, B., & D’Apuzzo, L. (2010). Characterizations of consistent pairwise comparison matrices over abelian linearly ordered groups. International Journal of Intelligent Systems, 25(10), 1035–1059.

    Article  Google Scholar 

  • Choo, E., & Wedley, W. (2004). A common framework for deriving preference values from pairwise comparison matrices. Computers & Operations Research, 31(6), 893–908.

    Article  Google Scholar 

  • Chu, A. T. W., Kalaba, R. E., & Springarn, K. (1979). A comparison of two methods for determining the weights of belonging to fuzzy sets. Journal of Optimization Theory and Applications, 27(4), 531–538.

    Article  Google Scholar 

  • Cohen, J. (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.

    Article  Google Scholar 

  • Crawford, G., & Williams, C. (1985). A note on the analysis of subjective judgement matrices. Journal of Mathematical Psychology, 29(4), 25–40.

    Article  Google Scholar 

  • Duszak, Z., & Koczkodaj, W. W. (1994). Generalization of a new definition of consistency for pairwise comparisons. Information Processing Letters, 52(5), 273–276.

    Article  Google Scholar 

  • Fedrizzi, M., & Brunelli, M. (2009). Fair consistency evaluation for reciprocal relations and in group decision making. New Mathematics and Natural Computation, 5(2), 407–420.

    Article  Google Scholar 

  • Golden, B. L., & Wang, Q. (1989). An alternate measure of consistency. In B. L. Golden, E. A. Wasil, & P. T. Harker (Eds.), The analytic hierarchy process, applications and studies (pp. 68–81). Berlin: Springer.

    Chapter  Google Scholar 

  • Karelaia, N., & Hogarth, R. M. (2008). Determinants of linear judgment: a meta-analysis of lens studies. Psychological Bulletin, 134(3), 404–426.

    Article  Google Scholar 

  • Koczkodaj, W. W. (1993). A new definition of consistency of pairwise comparisons. Mathematical and Computer Modelling, 18(7), 79–84.

    Article  Google Scholar 

  • Linares, P. (2009). Are inconsistent decisions better? An experiment with pairwise comparisons. European Journal of Operational Research, 193(2), 492–498.

    Article  Google Scholar 

  • Miller, G. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63(2), 81–97.

    Article  Google Scholar 

  • Osei-Bryson, K.-M. (2006). An action learning approach for assessing the consistency of pairwise comparison data. European Journal of Operational Research, 174(1), 234–244.

    Article  Google Scholar 

  • Pelàez, J. I., & Lamata, M. T. (2003). A new measure of consistency for positive reciprocal matrices. Computers & Mathematics with Applications, 46(12), 1839–1845.

    Article  Google Scholar 

  • Phelps, R. H., & Shanteau, J. (1978). Livestock judges: how much information can an expert use? Organizational Behavior and Human Performance, 21(2), 209–219.

    Article  Google Scholar 

  • Ramík, J., & Korviny, P. (2010). Inconsistency of pair-wise comparison matrix with fuzzy elements based on the geometric mean. Fuzzy Sets and Systems, 161(1), 1604–1613.

    Article  Google Scholar 

  • Ramík, J., & Perzina, R. (2010). A method for solving fuzzy multicriteria decision problems with dependent criteria. Fuzzy Optimization and Decision Making, 9(2), 123–141.

    Article  Google Scholar 

  • Saaty, T. L. (1974). Measuring the fuzziness of sets. Journal of Cybernetics, 4(4), 53–61.

    Article  Google Scholar 

  • Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234–281.

    Article  Google Scholar 

  • Saaty, T. L. (1980). The analytical hierarchy process. New York: McGraw-Hill.

    Google Scholar 

  • Saaty, T. L. (1994). Highlights and critical points in the theory and application of the analytic hierarchy process. European Journal of Operational Research, 74(3), 426–447.

    Article  Google Scholar 

  • Salo, A. A. (1993). Inconsistency analysis by approximately specified priorities. Mathematical and Computer Modelling, 17(4–5), 123–133.

    Article  Google Scholar 

  • Salo, A. A., & Hämäläinen, R. P. (1995). Preference programming through approximate ratio comparisons. European Journal of Operational Research, 82(3), 458–475.

    Article  Google Scholar 

  • Shanteau, J. (1989). Psychological characteristics and strategies of expert decision makers. In B. Rohrmann, L. R. Beach, C. Vleck, & S. R. Watson (Eds.), Advances in decision research (pp. 203–215). Amsterdam: North-Holland.

    Google Scholar 

  • Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. (2003). How can you tell if someone is an expert? Performance-based assessment of expertise. In S. L. Schneider & J. Shanteau (Eds.), Emerging perspectives on judgment and decision research (pp. 620–642). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Shiraishi, S., & Obata, T. (2002). On a maximization problem arising from a positive reciprocal matrix in the AHP. Bulletin of Informatics and Cybernetics, 34(2), 91–96.

    Google Scholar 

  • Shiraishi, S., Obata, T., & Daigo, M. (1998). Properties of a positive reciprocal matrix and their application to AHP. Journal of the Operations Research Society of Japan, 41(3), 404–414.

    Google Scholar 

  • Shiraishi, S., Obata, T., Daigo, M., & Nakajima, N. (1999). Assessment for an incomplete matrix and improvement of the inconsistent comparison: computational experiments. In Proceedings of ISAHP 1999, Kobe, Japan.

    Google Scholar 

  • Slovic, P. (1969). Analyzing the expert judge: a descriptive study of a stockbroker’s decision processes. Journal of Applied Psychology, 53(4), 255–263.

    Article  Google Scholar 

  • Snedecor, G. W., & Cochran, W. C. (1980). Statistical methods. Ames: Iowa State University Press.

    Google Scholar 

  • Spearman, C. E. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101.

    Article  Google Scholar 

  • Stein, W. E., & Mizzi, P. J. (2007). The harmonic consistency index for the analytic hierarchy process. European Journal of Operational Research, 177(1), 488–497.

    Article  Google Scholar 

  • Thurstone, L. L. (1927). A law of comparative judgement. Psychological Review, 34(4), 273–386.

    Article  Google Scholar 

  • Weiss, D. J., & Shanteau, J. (2003). Empirical assessment of expertise. Human Factors, 45(1), 104–114.

    Article  Google Scholar 

  • Weiss, D. J., Shanteau, J., & Harries, P. (2006). People who judge people. Journal of Behavioral Decision Making, 19(5), 441–454.

    Article  Google Scholar 

Download references

Acknowledgements

We are very grateful for the reviewers’ thorough and constructive comments.

Author information

Authors and Affiliations

  1. Systems Analysis Laboratory, Department of Mathematics and Systems Analysis, Aalto University, Espoo, Finland

    Matteo Brunelli

  2. Institute for Advanced Management Systems Research, Åbo Akademi University, Åbo, Finland

    Matteo Brunelli

  3. Department of Cognitive Sciences, University of Trento, Rovereto, Italy

    Luisa Canal

  4. Department of Industrial Engineering, University of Trento, Trento, Italy

    Michele Fedrizzi

Authors
  1. Matteo Brunelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luisa Canal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michele Fedrizzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matteo Brunelli.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Brunelli, M., Canal, L. & Fedrizzi, M. Inconsistency indices for pairwise comparison matrices: a numerical study. Ann Oper Res 211, 493–509 (2013). https://doi.org/10.1007/s10479-013-1329-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-013-1329-0

Keywords

Search

Navigation