Top

Journal of Business and Psychology

Published in:

14-03-2022 | Original Paper

Optimizing Measurement Reliability in Within-Person Research: Guidelines for Research Design and R Shiny Web Application Tools

Authors: Liu-Qin Yang, Wei Wang, Po-Hsien Huang, Anthony Nguyen

Published in: Journal of Business and Psychology | Issue 6/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Within-person research has become increasingly popular over recent years in the field of organizational studies for its unique theoretical and methodological advantages for studying dynamic intrapersonal processes (e.g., Dalal et al., Journal of Management 40:1396–1436, 2014; McCormick et al., Journal of Management 46:321–350, 2020). Despite the advancements, there remain serious challenges for many organizational researchers to fully appreciate and appropriately implement within-person research—more specifically, to correctly conceptualize and compute the within-person measurement reliability, as well as navigate key within-person research design factors (e.g., number of measurement occasions, T; number of participants, N; and scale length, I) to optimize within-person reliability. By conducting a comprehensive Monte Carlo simulation with 3240 data conditions, we offer a practical guideline table showing the expected within-person reliability as a function of key design factors. In addition, we provide three easy-to-use, free R Shiny web applications for within-person researchers to conveniently (a) compute expected within-person reliability based on their customized research design, (b) compute observed validity based on the expected reliability and hypothesized within-person validity, and (c) compute observed within-person (as well as between-person) reliability from collected within-person research datasets. We hope these much-needed evidence-based guidelines and practical tools will help enhance within-person research in organizational studies.

previous article Cognitive Ability and Job Performance: Meta-analytic Evidence for the Validity of Narrow Cognitive Abilities

next article Granting Leadership to Asian Americans: the Activation of Ideal Leader and Ideal Follower Traits on Observers’ Leadership Perceptions

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

The R shiny web applications are available through URL https://psychmethods.shinyapps.io/WithinPersonResearch or https://tinyurl.com/LevelAlpha.

Our research team also conducted a separate set of simulations using realistic research design conditions based on the empirical review of 103 unique organizational ESM studies published in 10 representative organizational research journals (e.g., scale length of 2–5 items, 10 vs. 25 measurement occasions, 60 vs. 90 vs. 120 participants), to compare the averaged biases and root mean squared errors (RMSE) of the alpha and omega reliability estimates relative to the true reliabilities. We found that the alpha and omega methods were equally efficacious in terms of having little biases and RMSE in estimating within- and between-person reliabilities for unidimensional scales.

Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Waveland Press.

Bakker, A. B., Sanz-Vergel, A. I., Rodríguez-Muñoz, A., & Oerlemans, W. G. M. (2015). The state version of the recovery experience questionnaire: A multilevel confirmatory factor analysis. European Journal of Work and Organizational Psychology, 24(3), 350–359. https://doi.org/10.1080/1359432X.2014.903242CrossRef

Beal, D. J. (2015). ESM 2.0: State of the art and future potential of experience sampling methods in organizational research. Annual Review of Organizational Psychology and Organizational Behavior, 2(1), 383–407. https://doi.org/10.1146/annurev-orgpsych-032414-111335CrossRef

Beal, D. J., & Weiss, H. M. (2003). Methods of ecological momentary assessment in organizational research. Organizational Research Methods, 6(4), 440–464. https://doi.org/10.1177/1094428103257361CrossRef

Bliese, P. D., Chan, D., & Ployhart, R. E. (2007). Multilevel methods: Future directions in measurement, longitudinal analyses, and nonnormal outcomes. Organizational Research Methods, 10(4), 551–563. https://doi.org/10.1177/1094428107301102CrossRef

Brose, A., Voelkle, M. C., Lövdén, M., Lindenberger, U., & Schmiedek, F. (2015). Differences in the between–person and within–person structures of affect are a matter of degree. European Journal of Personality, 29(1), 55–71. https://doi.org/10.1002/per.1961CrossRef

Cho, E. (2016). Making reliability reliable: A systematic approach to reliability coefficients. Organizational Research Methods, 19(4), 651–682. https://doi.org/10.1177/1094428116656239CrossRef

Cortina, J. M., Sheng, Z., Keener, S. K., Keeler, K. R., Grubb, L. K., Schmitt, N., Tonidandel, S., Summerville, K. M., Heggestad, E. D., & Banks, G. C. (2020). From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology. Journal of Applied Psychology, 105(12), 1351–1381.CrossRef

Cranford, J. A., Shrout, P. E., Iida, M., Rafaeli, E., Yip, T., & Bolger, N. (2006). A procedure for evaluating sensitivity to within-person change: Can mood measures in diary studies detect change reliably? Personality and Social Psychology Bulletin, 32(7), 917–929. https://doi.org/10.1177/0146167206287721CrossRefPubMed

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi-org.proxy.lib.pdx.edu/https://doi.org/10.1007/BF02310555

Curran, P. J., & Bauer, D. J. (2011). The disaggregation of within-person and between-person effects in longitudinal models of change. Annual Review of Psychology, 62(1), 583–619. https://doi.org/10.1146/annurev.psych.093008.100356CrossRefPubMedPubMedCentral

Dalal, R. S., Bhave, D. P., & Fiset, J. (2014). Within-person variability in job performance. Journal of Management, 40(5), 1396–1436. https://doi.org/10.1177/0149206314532691CrossRef

DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 9(3), 327–346. https://doi.org/10.1207/S15328007SEM0903_2

Ellis, A. M., Bauer, T. N., Mansfield, L. R., Erdogan, B., Truxillo, D. M., & Simon, L. S. (2015). Navigating uncharted waters: Newcomer socialization through the lens of stress theory. Journal of Management, 41(1), 203–235. https://doi.org/10.1177/0149206314557525CrossRef

Ferrer, E., & McArdle, J. J. (2010). Longitudinal modeling of developmental changes in psychological research. Current Directions in Psychological Science, 19(3), 149–154. https://doi.org/10.1177/0963721410370300CrossRef

Fisher, C. D., & To, M. L. (2012). Using experience sampling methodology in organizational behavior. Journal of Organizational Behavior, 33(7), 865–877. https://doi.org/10.1002/job.1803CrossRef

Fisher, G. G., Matthews, R. A., & Gibbons, A. M. (2016). Developing and investigating the use of single-item measures in organizational research. Journal of Occupational Health Psychology, 21(1), 3–23. https://doi.org/10.1037/a0039139CrossRefPubMed

Fleeson, W. (2001). Toward a structure- and process integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology, 80(6), 1011–1027. https://doi.org/10.1037/0022-3514.80.6.1011CrossRefPubMed

Gabriel, A. S., Podsakoff, N. P., Beal, D. J., Scott, B. A., Sonnentag, S., Trougakos, J. P., & Butts, M. M. (2019). Experience sampling methods: A discussion of critical trends and considerations for scholarly advancement. Organizational Research Methods, 22(4), 969–1006. https://doi.org/10.1177/1094428118802626CrossRef

Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multi-level confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138CrossRefPubMed

Huang, P.-H., & Weng, L.-J. (2012). Estimating the reliability of aggregated and within-person centered scores in ecological momentary assessment. Multivariate Behavioral Research, 47(3), 421–441. https://doi.org/10.1080/00273171.2012.673924CrossRefPubMed

Hox, J. (2002). Multi-level analysis: Techniques and applications. Lawrence Erlbaum Associates.

Logg, J. M., & Dorison, C. A. (2021). Pre-registration: Weighing costs and benefits for researchers. Organizational Behavior and Human Decision Processes, 167, 18–27. https://doi.org/10.1016/j.obhdp.2021.05.006CrossRef

McCormick, B. W., Reeves, C. J., Downes, P. E., Li, N., & Ilies, R. (2020). Scientific contributions of within-person research in management: Making the juice worth the squeeze. Journal of Management, 46(2), 321–350. https://doi.org/10.1177/0149206318788435CrossRef

McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates Publishers.

Meredith, W., & Tisak, J. (1990). Latent curve analysis. Psychometrika, 55 (1), 107–122. https://doi-org.proxy.lib.pdx.edu/https://doi.org/10.1007/BF02294746

Neale, M. C., Hunter, M. D., Pritikin, J. N., Zahery, M., Brick, T. R., Kirkpatrick, R. M., et al. (2016). OpenMx 2.0: Extended structural equation and statistical modeling. Psychometrika, 81, 535–549. https://doi.org/10.1007/s11336-014-9435-8CrossRefPubMed

Nezlek, J. B. (2017). A practical guide to understanding reliability in studies of within-person variability. Journal of Research in Personality, 69, 149–155. https://doi.org/10.1016/j.jrp.2016.06.020CrossRef

Nguyen, A. N., Yang, L.-Q., Wang, W., & Huang, P.-H. (November 2019). Is your diary “Reliable”?: A review of current measurement practices in ESM research. Paper presented at the Biennnial conference of Conference of Work, Stress, and Health, Philadelphia, PA.

Ohly, S., Sonnentag, S., Niessen, C., & Zapf, D. (2010). Diary studies in organizational research. Journal of Personnel Psychology, 9(2), 79–93. https://doi.org/10.1027/1866-5888/a000009CrossRef

Ployhart, R. E., & Vandenberg, R. J. (2010). Longitudinal research: The theory, design, and analysis of change. Journal of Management, 36(1), 94–120. https://doi.org/10.1177/0149206309352110CrossRef

Raudenbush, S. W., Rowan, B., & Kang, S. J. (1991). A multilevel, multivariate model for studying school climate with estimation via the EM algorithm and application to US high-school data. Journal of Educational Statistics, 16(4), 295–330. https://doi.org/10.2307/1165105CrossRef

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/. Accessed 3/8/2020

Reis, H. T., & Gable, S. L. (2000). Event-sampling and other methods for studying everyday experience. In H. T. [Ed Reis & C. M. [Ed Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 190–222, Chapter xii, 558 Pages). Cambridge University Press (New York, NY, US).

Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315

Robinson, M. D., & Clore, G. L. (2002). Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin, 128(6), 934.CrossRef

Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach to the measurement of change. Psychological Bulletin, 92(3), 726–748. http://dx.doi.org.proxy.lib.pdx.edu/https://doi.org/10.1037/0033-2909.92.3.726

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Miufflin Company.

Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research, 7(3), 301–317. https://doi.org/10.1177/096228029800700306CrossRefPubMed

Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15, 72–101. https://doi.org/10.2307/1422689CrossRef

Tay, L., Woo, S. E., & Vermunt, J. K. (2014). A conceptual and methodological framework for psychometric isomorphism: Validation of multilevel construct measures. Organizational Research Methods, 17(1), 77–106. https://doi.org/10.1177/1094428113517008CrossRef

Wanous, J. P., & Reichers, A. E. (1996). Estimating the reliability of a single-item measure. Psychological Reports, 78(2), 631–634. https://doi.org/10.2466/pr0.1996.78.2.631CrossRef

Wänström, L. (2009). Sample sizes for two-group second-order latent growth curve models. Multivariate Behavioral Research, 44(5), 588–619. https://doi.org/10.1080/00273170903202589CrossRef

Willett, J. B. (1989). Some results on reliability for the longitudinal measurement of change: Implications for the design of studies of individual growth. Educational and Psychological Measurement, 49 (3), 587- 602. https://doi-org.proxy.lib.pdx.edu/https://doi.org/10.1177/001316448904900309

Zapf, D., Dormann, C., & Frese, M. (1996). Longitudinal studies in organizational stress research: A review of the literature with reference to methodological issues. Journal of Occupational Health Psychology, 1(2), 145–169. https://doi.org/10.1037/1076-8998.1.2.145CrossRefPubMed

Title: Optimizing Measurement Reliability in Within-Person Research: Guidelines for Research Design and R Shiny Web Application Tools
Authors: Liu-Qin Yang
Wei Wang
Po-Hsien Huang
Anthony Nguyen
Publication date: 14-03-2022
Publisher: Springer US
Published in: Journal of Business and Psychology / Issue 6/2022
Print ISSN: 0889-3268
Electronic ISSN: 1573-353X
DOI: https://doi.org/10.1007/s10869-022-09803-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 6/2022

Incivility Begets Incivility: Understanding the Relationship Between Experienced and Enacted Incivility with Customers Over Time

Ever Thought About Strikes? Development of a Scale to Assess Attitudes and Behavioral Reactions to Strikes

When a Good Day Makes for a Good Night: Antecedents of Sleep and Morning States

Beyond the Baby Bump: Subtle Discrimination Against Working Mothers In the Hiring Process

Cognitive Ability and Job Performance: Meta-analytic Evidence for the Validity of Narrow Cognitive Abilities

Opening the Black Box of the Response Process to Personality Faking: An Application of Item Response Tree Models