Skip to main content
SpringerLink
Log in

Assessing the maturity of a research area: bibliometric review and proposed framework

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In recent years, many disciplines have begun to adopt more systematic and standardized approaches to evaluate the impact and development of a research area with a stronger emphasis on quantitative techniques. In particular, identifying and analyzing the published literature have become important exercises for many disciplines and methods such as systematic literature review and bibliometric analysis have become more regularly used to obtain a deeper understanding of a research area. One concept that is of particular interest is the maturity, or level of development, of a research area. While this concept has been mentioned in many works, it has not yet been formalized, resulting in a lack of consensus concerning the definition of research area maturity and analysis techniques to assess maturity. Therefore, most assessments of research area maturity consider only a subset of the possible criteria with significant differences in the metrics and analyses used among different disciplines. Due to the inconsistencies in the definition and assessment of this concept, a comprehensive synthesis of this literature area is needed. This paper presents the results of a study to identify and analyze the literature, define the maturity of a research area, and synthesize the criteria for assessing maturity. The results are used to develop a generalized maturity assessment framework that establishes a comprehensive set of criteria, which can be adapted for use across a variety of research areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Archambault, É., & Larivière, V. (2010). The limits of bibliometrics for the analysis of the social sciences and humanities literature. World Social Science Report, 251–254.

  • Becheikh, N. (2010). How to improve knowledge transfer strategies and practices in education? Answers from a systematic literature review. Research in Higher Education Journal, 7, 1–21.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. doi:10.1002/9780470743386.

  • Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. doi:10.1007/s11192-015-1565-y.

    Article  Google Scholar 

  • Borrego, M., & Bernhard, J. (2011). The emergence of engineering education research as an internationally connected field of inquiry. Journal of Engineering Education, 100(1), 14–47. doi:10.1002/j.2168-9830.2011.tb00003.x.

    Article  Google Scholar 

  • Brookes, B. C. (1990). Biblio-, sciento-, infor-metrics? What are we talking about? In L. Egghe, & R. Rousseau (Eds.), Informetrics 89/90 (pp. 31–43). Diepenbeek.

  • Budi, I., Aji, R. F., & Widodo, A. (2013). Prediction of research topics on science & technology (S&T) using ensemble forecasting. International Journal of Software Engineering and Its Applications, 7(5), 253–268. doi:10.14257/ijseia.2013.7.5.23.

    Article  Google Scholar 

  • Campbell Resource Center. (n.d.). Retrieved January 23, 2015, from http://www.campbellcollaboration.org/resources/resource_center.php.

  • Carlile, P. R., & Christensen, C. M. (2004). The cycles of theory building in management research, version 5.0, Harvard Business School Working Knowledge. http://www.hbs.edu/rcscarch/pdf/05-057.pdf. Accessed 21 March 2011.

  • Cheon, M. J., Groven, V., & Sabherwal, R. (1993). The evolution of empirical research in IS. Information & Management, 24(3), 107–119. doi:10.1016/0378-7206(93)90060-7.

    Article  Google Scholar 

  • Cobo, M. J., López-Herrera, A. G., Herrera-Viedma, E., & Herrera, F. (2011). An approach for detecting, quantifying, and visualizing the evolution of a research field: A practical application to the fuzzy sets theory field. Journal of Informetrics, 5(1), 146–166. doi:10.1016/j.joi.2010.10.002.

    Article  MATH  Google Scholar 

  • Cooper, H., & Hedges, L. V. (2009). Research synthesis as a scientific process. In The handbook of research synthesis and meta-analysis (pp. 3–17). Retrieved from http://books.google.com/books?hl=en&lr=&id=LUGd6B9eyc4C&oi=fnd&pg=PA3&dq=RESEARCH+SYNTHESIS+AS+A+SCIENTIFIC+PROCESS&ots=5MEJyT_n6P&sig=QS5-Jcd7-Wzvo8pxzZD0ktb33SM.

  • Cooper, H., Hedges, L., & Valentine, J. (2009). The handbook of research synthesis and meta-analysis. Psychological Review April 2006 (Vol. 113).

  • Cronin, P., Ryan, F., Coughlan, M., Reid, E. R., Harris, B., Vessali, K. V., et al. (2007). Undertaking a literature review: A step-by-step approach. Computers, Environment and Urban Systems, 60(1), 1–15. doi:10.1177/107808747000500401.

    Google Scholar 

  • Diallo, S. Y., Lynch, C. J., Gore, R., & Padilla, J. J. (2016). Identifying key papers within a journal via network centrality measures. Scientometrics, 107(3), 1–16. doi:10.1007/s11192-016-1891-8.

    Article  Google Scholar 

  • Drew, C. H., Pettibone, K. G., Finch, F. O., Giles, D., & Jordan, P. (2016). Automated research impact assessment: A new bibliometrics approach. Scientometrics, 106(3), 987–1005. doi:10.1007/s11192-015-1828-7.

    Article  Google Scholar 

  • Gagnon, R. J., & Ghosh, S. (1991). Assembly line research: Historical roots, research life cycles and future directions. Omega, 19(5), 381–399. doi:10.1016/0305-0483(91)90056-Y.

    Article  Google Scholar 

  • Garfield, E. (2006). Citation indexes for science. A new dimension in documentation through association of ideas. International Journal of Epidemiology, 35(5), 1123–1127. doi:10.1093/ije/dyl189.

    Article  Google Scholar 

  • Garfield, E. (2009). From the science of science to scientometrics visualizing the history of science with HistCite software. Journal of Informetrics, 3(3), 173–179.

    Article  Google Scholar 

  • Grover, V. (2012). The information systems field: Making a case for maturity and contribution. Journal of the Association for Information Systems, 13(Special Issue), 254–272.

    MathSciNet  Google Scholar 

  • Harvey, L. J., & Myers, M. D. (1995). Scholarship and practice: The contribution of ethnographic research methods to bridging the gap. Information Technology & People, 8(3), 13–27. doi:10.1108/09593849510098244.

    Article  Google Scholar 

  • Hicks, D. (1999). The difficulty of achieving full coverage of international social science literature and the bibliometric consequences. Scientometrics, 44(2), 193–215.

    Article  Google Scholar 

  • Higgins, J., & Green, S. (Eds.). (2011). Chapter 4: Guide to the contents of a Cochrane protocol and review. Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration. doi:10.1002/9780470712184.

  • Hood, W. W., & Wilson, C. S. (2001). The literature of bibliometrics and informetrics scientometrics. Scientometrics, 52(2), 291–314.

    Article  Google Scholar 

  • Houy, C., Fettke, P., & Loos, P. (2010). Empirical research in business process management—Analysis of an emerging field of research. Business Process Management Journal, 16(4), 619–661. doi:10.1108/14637151011065946.

    Article  Google Scholar 

  • Karuga, G., Lowry, P., & Richardson, V. (2007). Assessing the impact of premier information systems research over time. Communications of the Association for Information Systems, 19, 115–131. Retrieved from http://aisel.aisnet.org/cgi/viewcontent.cgi?article=2650&context=cais.

  • Keathley, H., Bean, A., Chen, T., Vila, K., Ye, K., & Gonzalez-Aleu, F. (2015). Bibliometric analysis of author collaboration in engineering management research. In Proceedings of the 2015 international annual conference, American Society for Engineering Management, October 7-10, 2015.

  • Keathley, H., Bean, A., Chen, T., Vila, K., Ye, K., & Gonzalez-Aleu, F. (2016). Bibliometric analysis of research design characteristics in engineering management research. In Proceedings of the 2016 industrial and systems engineering research conference, Institute of Industrial Engineers, May 21-24, 2016.

  • Li, X., Hu, D., Dang, Y., Chen, H., Roco, M. C., Larson, C. A., et al. (2009). Nano Mapper: An Internet knowledge mapping system for nanotechnology development. Journal of Nanoparticle Research, 11(3), 529–552. doi:10.1007/s11051-008-9491-z.

    Article  Google Scholar 

  • Lim, J., Rong, G., & Grover, V. (2007). An inductive approach to documenting the “core” and evolution of the IS field. Communications of the Association for Information Systems, 19(32), 665–691. Retrieved from http://aisel.aisnet.org/cgi/viewcontent.cgi?article=2675&context=cais.

  • Maloni, M. J., Carter, C. R., & Carr, A. S. (2009). Assessing logistics maturation through author concentration. International Journal of Physical Distribution & Logistics Management, 39(3), 250–268. doi:10.1108/09600030910951728.

    Article  Google Scholar 

  • Maloni, M., Carter, C. R., & Kaufmann, L. (2012). Author affiliation in supply chain management and logistics journals: 2008–2010. International Journal of Physical Distribution & Logistics Management,. doi:10.1108/09600031211202481.

    Google Scholar 

  • Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. doi:10.1007/s11192-015-1765-5.

    Article  Google Scholar 

  • Moody, D. L. (2000). Building links between IS research and professional practice: Improving the relevance and impact of IS research. In Proceedings of the twenty first international conference on information systems (Issue 1, pp. 351–360). Retrieved from http://dl.acm.org/citation.cfm?id=359640.359760.

  • Neely, A. (2005). The evolution of performance measurement research: Developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12), 1264–1277. doi:10.1108/01443570510633648.

    Article  Google Scholar 

  • Nie, K., Ma, T., & Nakamori, Y. (2009). An approach to aid understanding emerging research fields—The case of knowledge management. Systems Research and Behavioral Science, 26(6), 629–643. doi:10.1002/sres.926.

    Article  Google Scholar 

  • Nissen, M. E. (1995). A focused review or the reengineering literature: Expert frequently asked questions. Quality Management Journal, 3(3), 52–66.

    Google Scholar 

  • Pasqualine, A., Plytiuk, C. F., da Costa, S. E. G., & de Lima, E. P. (2012). Performance management in healthcare: A bibliometric review. In IIE Annual Conference. Proceedings (p. 1). Institute of Industrial Engineers-Publisher.

  • Patra, S. K., Bhattacharya, P., & Verma, N. (2006). Bibliometric study of literature on bibliometrics. DESIDOC Bulletin of Information Technology, 26(1), 27–32. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=23638785&site=ehost-live.

  • Paul-Hus, A., Desrochers, N., & Costas, R. (2016). Characterization, description, and considerations for the use of funding acknowledgement data in Web of Science. Scientometrics,. doi:10.1007/s11192-016-1953-y.

    Google Scholar 

  • Perianes-Rodriguez, A., & Ruiz-Castillo, J. (2016). A comparison of two ways of evaluating research units working in different scientific fields. Scientometrics, 106(2), 539–561. doi:10.1007/s11192-015-1801-5.

    Article  Google Scholar 

  • Piro, F. N., Rørstad, K., & Aksnes, D. W. (2016). How does prolific professors influence on the citation impact of their university departments? Scientometrics, 107(3), 1–21. doi:10.1007/s11192-016-1900-y.

    Article  Google Scholar 

  • Porter, A. L., & Detampel, M. J. (1995). Technology opportunities analysis. Technological Forecasting and Social Change, 49(3), 237–255. doi:10.1016/0040-1625(95)00022-3.

    Article  Google Scholar 

  • Pritchard, A. (1969). Statistical bibliography or bibliometrics. Journal of Documentation, 25, 348.

    Google Scholar 

  • Schmenner, R. W., Wassenhove, L. Van, Ketokivi, M., Heyl, J., & Lusch, R. F. (2009). Too much theory, not enough understanding. Journal of Operations Management, 27(5), 339–343. doi:10.1016/j.jom.2009.07.004.

    Article  Google Scholar 

  • Schoepflin, U., & Glänzel, W. (2001). Two decades of “Scientometrics” An interdisciplinary field represented by its leading journal. Scientometrics, 50(2), 301–312. doi:10.1023/A:1010577824449.

    Article  Google Scholar 

  • Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4), 265–269.

    Article  Google Scholar 

  • Small, H. (1999). Visualizing science by citation mapping. Journal of the American Society for Information Science, 50(9), 799–813. doi:10.1002/(SICI)1097-4571(1999)50:9<799:AID-ASI9>3.0.CO;2-G.

    Article  Google Scholar 

  • Smith, D. R. (2012). Impact factors, scientometrics and the history of citation-based research. Scientometrics, 92(2), 419–427.

    Article  Google Scholar 

  • Stone, K. B. (2012). Four decades of lean: A systematic literature review. International Journal of Lean Six Sigma, 3(2), 112–132. doi:10.1108/20401461211243702.

    Article  Google Scholar 

  • Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. doi:10.1007/s11192-013-1117-2.

    Article  Google Scholar 

  • Sun, X., Lin, H., Xu, K., & Ding, K. (2015). How we collaborate: Characterizing, modeling and predicting scientific collaborations. Scientometrics,. doi:10.1007/s11192-015-1597-3.

    Google Scholar 

  • Taylor, A., & Taylor, M. (2009). Operations management research: Contemporary themes, trends and potential future directions. International Journal of Operations & Production Management, 29(12), 1316–1340. doi:10.1108/01443570911006018.

    Article  Google Scholar 

  • Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14, 207–222. doi:10.1111/1467-8551.00375.

    Article  Google Scholar 

  • Walsh, D., & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Journal of Advanced Nursing,. doi:10.1111/j.1365-2648.2005.03380.x.

    Google Scholar 

  • Wendler, R. (2012). The maturity of maturity model research: A systematic mapping study. Information and Software Technology, 54(12), 1317–1339. doi:10.1016/j.infsof.2012.07.007.

    Article  Google Scholar 

  • Zhao, Y., & Zhao, R. (2016). An evolutionary analysis of collaboration networks in scientometrics. Scientometrics, 107(2), 759–772. doi:10.1007/s11192-016-1857-x.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL, USA

    Heather Keathley-Herring

  2. Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA, USA

    Eileen Van Aken & Pablo Cardenas Orlandini

  3. Department of Engineering, Universidad de Monterrey, San Pedro Garza Garcia, Mexico

    Fernando Gonzalez-Aleu

  4. Polytechnic School, Pontifical Catholic University of Parana, Curitiba, Brazil

    Fernando Deschamps

  5. Department of Mechanical Engineering, Federal University of Parana, Curitiba, Brazil

    Fernando Deschamps

  6. Department of Economics, Management and Leadership, Royal Military Academy, Brussels, Belgium

    Geert Letens

Authors
  1. Heather Keathley-Herring
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eileen Van Aken
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fernando Gonzalez-Aleu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fernando Deschamps
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Geert Letens
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pablo Cardenas Orlandini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heather Keathley-Herring.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Keathley-Herring, H., Van Aken, E., Gonzalez-Aleu, F. et al. Assessing the maturity of a research area: bibliometric review and proposed framework. Scientometrics 109, 927–951 (2016). https://doi.org/10.1007/s11192-016-2096-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-2096-x

Keywords

Search

Navigation