nach oben

Journal of Business and Psychology

Erschienen in:

26.07.2016 | Original Paper

Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients

verfasst von: Maura I. Burke, Ronald S. Landis, Michael J. Burke

Erschienen in: Journal of Business and Psychology | Ausgabe 6/2017

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Purpose

In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.

Design/Methodology/Approach

Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.

Findings

Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.

Implications

Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.

Originality/Value

This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.

Nächster Artikel How Far Can Support Go? Supported Supervisors’ Performance and Subordinate Dedication

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Equations used have been reported elsewhere as noted. Numbering of the equations in the current paper is for ease of presentation only.

The RBNL meta-analytic procedures for correcting individual correlations depart from those proposed by Hunter and Schmidt (2004) by accounting not only for sampling error due to N, but additionally take into account sampling error in the reliability estimates. As discussed in Raju et al. (1991), Hunter and Schmidt’s (2004) meta-analytic procedures for individually corrected correlations deal with the special case in which reliabilities of predictor and criterion are assumed to be fixed. The reader is referred to Raju et al. (1991) for a detailed description concerning the advantages of employing sample-based meta-analytic procedures over alternative methods.

One study was excluded from the analyses due to a large sample size and the direction of the effect. Results of these analyses with the study included are available upon request from the author.

Supplemental meta-analyses were also conducted with a 75 % reduction in available reliability from the original amount and are available upon request from the first author. Most notably, when sample-based reliabilities (and their associated sampling variances) are artificially extracted we see a marked increase in \({\text{SE}}_{{M_{\rho } }}\) and a substantially larger magnitude of M _ρ. Furthermore, the range of confidence moves farther away from original surrounding calculations when the data naturally occurred.

*Studies included in meta-analytic calculations

Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917–1992. Journal of Applied Psychology, 77, 836–874. doi:10.1037/0021-9010.77.6.836.CrossRef

*Baer, M., & Frese, M. (2003). Innovation is not enough: Climates for initiative and psychological safety, process innovations, and firm performance. Journal of Organizational Behavior, 24, 45–68. doi:10.1002/job.179.CrossRef

Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765. doi:10.1037/0033-2909.83.5.762.CrossRef

Bliese, P. D. (2000). Within group-agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.

*Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943–962. doi:10.1002/(SICI)1099-1379(199911)20:6<943::AID-JOB976>3.0.CO;2-9.CrossRef

Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in conducting and interpreting meta-analyses. In K. Murphy (Ed.), Validity generalization: A critical review (pp. 287–309). Mahwah, NJ: Erlbaum.

Burke, M. J., Landis, R. J., & Burke, M. I. (2014). 80 and beyond: Recommendations for disattenuating correlations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 531–535.

*Chen, G., Kirkman, B., Kanfer, R., Allen, D., & Rosen, B. (2007). A multilevel study of leadership, empowerment, and performance in teams. Journal of Applied Psychology, 92, 331–346. doi:10.1037/0021-9010.92.2.331.CrossRefPubMed

Christian, M. S., Bradley, J. C., Wallace, J., & Burke, M. J. (2009). Workplace safety: A meta-analysis of the roles of person and situation factors. Journal of Applied Psychology, 94(5), 1103–1127. doi:10.1037/a0016172.CrossRefPubMed

*Collins, C. J., & Smith, K. G. (2006). Knowledge exchange and combination: The role of human resource practices in the performance of high-technology firms. Academy of Management Journal, 49, 544–560. doi:10.5465/AMJ.2006.21794671.CrossRef

*Colquitt, J. A., Noe, R. A., & Jackson, C. L. (2002). Justice in teams: Antecedents and consequences of procedural justice climate. Personnel Psychology, 55, 83–109. doi:10.1111/j.1744-6570.2002.tb00104.x.CrossRef

Cronbach, L. J., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.

*Dawson, J. F., Gonzalez-Roma, V., Davis, A., & West, M. A. (2008). Organisational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89–111. doi:10.1080/13594320601046664.CrossRef

*de Jong, A., de Ruyter, K., & Lemmink, J. (2005). Service climate in self-managing teams: Mapping the linkage of team member perceptions and service performance outcomes in a business-to-business setting. Journal of Management Studies, 42, 1593–1620. doi:10.1111/j.1467-6486.2005.00558.x.CrossRef

*Dietz, J., Pugh, S. D., & Wiley, J. (2004). Service climate effects on customer attitudes: An examination of boundary conditions. Academy of Management Journal, 47, 81–92. doi:10.2307/20159561.CrossRef

Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407–424. doi:10.1007/BF02288803.CrossRef

*Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61–94. doi:10.1111/j.1744-6570.2004.tb02484.x.CrossRef

*Fay, D., Luhrmann, H., & Kohl, C. (2004). Proactive climate in a post-reorganization setting: When staff compensate managers’ weakness. European Journal of Work and Organizational Psychology, 13, 241–267. doi:10.1080/13594320444000083.CrossRef

*Gelade, G. A., & Ivery, M. (2003). The impact of human resource management and work climate on organizational performance. Personnel Psychology, 56, 383–404. doi:10.1111/j.1744-6570.2003.tb00155.x.CrossRef

*Gelade, G. A., & Young, S. (2005). Test of a service profit chain model in the retail banking sector. Journal of Occupational & Organizational Psychology, 78, 1–22. doi:10.1348/096317904X22926.CrossRef

*Gil, F., Rico, R., Alcover, C. M., & Barrasa, A. (2005). Change-oriented leadership, satisfaction and performance in work groups. Journal of Managerial Psychology, 20, 312–328. doi:10.1108/02683940510589073.CrossRef

*Gonzalez-Roma, V., Fortes-Ferreira, L., & Peiro, J. (2009). Team climate, climate strength and team performance. A longitudinal study. Journal of Occupational and Organizational Psychology, 82, 511–536. doi:10.1348/096317908X370025.CrossRef

Harter, J. K., Schmidt, F. L., Asplund, J. W., Killham, E. A., & Agrawal, S. (2010). Causal impact of employee work perceptions on the bottom line of organizations. Perspectives on Psychological Science, 5(4), 378–389. doi:10.1177/1745691610374589.CrossRefPubMed

Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87(2), 268–279. doi:10.1037/0021-9010.87.2.268.CrossRefPubMed

*Hofmann, D. A., & Mark, B. (2006). An investigation of the relationship between safety climate and medication errors as well as other nurse and patient outcomes. Personnel Psychology, 59, 847–869. doi:10.1111/j.1744-6570.2006.00056.x.CrossRef

*Hofmann, D. A., & Stetzer, A. (1996). A cross-level investigation of factors influencing unsafe behaviors and accidents. Personnel Psychology, 49, 307–339. doi:10.1111/j.1744-6570.1996.tb01802.x.CrossRef

Hong, Y., Liao, H., Hu, J., & Jiang, K. (2013). Missing link in the service profit chain: A meta-analytic review of the antecedents, consequences, and moderators of service climate. Journal of Applied Psychology, 98(2), 237–267. doi:10.1037/a0031666.CrossRefPubMed

*Hui, C. H., Chiu, W. C. K., Yu, P. L. H., Cheng, K., & Tse, H. (2007). The effects of service climate and the effective leadership behaviour of supervisors on frontline employee service quality: A multi-level analysis. Journal of Occupational & Organizational Psychology, 80, 151–172. doi:10.1348/096317905X89391.CrossRef

Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.CrossRef

*Katz-Navon, T., Naveh, E., & Stern, Z. (2005). Safety climate in healthcare organizations: A multidimensional approach. Academy of Management Journal, 48, 1075–1089. doi:10.5465/AMJ.2005.19573110.CrossRef

*King, E. B., de Chermont, K., West, M., Dawson, J. F., & Hebl, M. (2007). How innovation can alleviate negative consequences of demanding work contexts: The influence of climate for innovation on organizational outcomes. Journal of Occupational and Organizational Psychology, 80, 631–645. doi:10.1348/096317906X171145.CrossRef

Lahey, M., Downey, R. G., & Saal, F. E. (1983). Intraclass correlations: There’s more there than meets the eye. Psychological Bulletin, 93(3), 586–595. doi:10.1037/0033-2909.93.3.586.CrossRef

LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 74, 478–500.

*Liao, H., & Chuang, A. (2004). A multilevel investigation of factors influencing employee service performance and customer outcomes. Academy of Management Journal, 47, 41–58. doi:10.2307/20159559.CrossRef

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Menlo Park, CA: Addison-Wesley.

*Luria, G. (2008). Controlling for quality: Climate, leadership, and behavior. Quality Management Journal, 15, 27–42.

*Luria, G., & Yagil, D. (2008). Procedural justice, ethical climate and service outcomes in restaurants. International Journal of Hospitality Management, 27, 276–283. doi:10.1016/j.ijhm.2007.07.022.CrossRef

*MacCormick, J. S., & Parker, S. K. (2010). A multiple climates approach to understanding business unit effectiveness. Human Relations, 63, 1171–1806. doi:10.1177/0018726710365090.

*Mayer, D. M., Kuenzi, M., & Greenbaum, R. L. (2010). Examining the link between ethical leadership and employee misconduct: The mediating role of ethical climate. Journal of Business Ethics, 95, 7–16. doi:10.1007/s10551-0110-794-0.CrossRef

*Maynard, M. T., Mathieu, J. E., Marsh, W. M., & Ruddy, T. M. (2007). A multilevel investigation of the influences of employees’ resistance to empowerment. Human Performance, 20, 147–171. doi:10.1080/08959280701332885.CrossRef

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30.CrossRef

*McKay, P. F., Avery, D. R., & Morris, M. A. (2009). A tale of two climates: Diversity climate from subordinates’ and managers’ perspectives and their role in store unit sales performance. Personnel Psychology, 62, 767–791. doi:10.1111/j.1744-6570.2009.01157.x.CrossRef

*Naumann, S., & Bennett, N. (2002). The effects of procedural justice climate on work group performance. Small Group Research, 33, 361–377. doi:10.1177/10496402033003004.CrossRef

*Naveh, E., Katz-Navon, T., & Stern, Z. (2005). Treatment errors in healthcare: A safety climate approach. Management Science, 51, 948–960. doi:10.1287/mnsc.1050.0372.CrossRef

*Neal, A., & Griffin, M. A. (2006). A study of the lagged relationships among safety climate, safety motivation, safety behavior, and accidents at the individual and group levels. Journal of Applied Psychology, 91, 946–953. doi:10.1037/0021-9010.91.4.946.CrossRefPubMed

*Ostroff, C., Kinicki, A., & Clark, M. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355–368. doi:10.1037/0021-9010.87.2.355.CrossRefPubMed

*Pirola-Merlo, A., Hartel, C., Mann, L., & Hirst, G. (2002). How leaders influence the impact of affective events on team climate and performance in R&D teams. Leadership Quarterly, 13, 561–581. doi:10.1016/S1048-9843(02)00144-3.CrossRef

Raju, N. S., & Brand, P. A. (2003). Determining the significance of correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 27, 52–71. doi:10.1177/0146621602239476.CrossRef

Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432–446. doi:10.1037/0021-9010.76.3.432.CrossRef

Raju, N. S., Pappas, S., & Williams, C. P. (1989). An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. Journal of Applied Psychology, 74, 901–911. doi:10.1037/0021-9010.74.6.901.CrossRef

*Richardson, H. A., & Vandenberg, R. J. (2005). Integrating managerial perceptions and transformational leadership into a work-unit level model of employee involvement. Journal of Organizational Behavior, 26, 561–589. doi:10.1002/job.329.CrossRef

*Salanova, M., Agut, S., & Peiro, J. M. (2005). Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology, 90, 1217–1227. doi:10.1037/0021-9010.90.6.1217.CrossRefPubMed

*Schmit, M. J., & Allscheid, S. P. (1995). Employee attitudes and customer satisfaction: Making theoretical and empirical connections. Personnel Psychology, 48, 521–536. doi:10.1111/j.1744-6570.1995.tb01768.x.CrossRef

*Schneider, B., Ehrhart, M. G., Mayer, D. M., Saltz, J. L., & Niles-Jolly, K. (2005). Understanding organization-customer links in service settings. Academy of Management Journal, 48, 1017–1032. doi:10.5465/AMJ.2005.19573107.CrossRef

*Schneider, B., White, S. S., & Paul, M. C. (1998). Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology, 83, 150–163. doi:10.1037/0021-9010.83.2.150.CrossRefPubMed

*Schulte, M., Ostroff, C., Shmulyian, S., & Kinicki, A. (2009). Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology, 3, 618–634. doi:10.1037/a0014365.CrossRef

*Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004). Taking empowerment to the next level: A multiple-level model of empowerment, performance, and satisfaction. Academy of Management Journal, 47, 332–349. doi:10.2307/20159585.CrossRef

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420.CrossRefPubMed

*Simons, T., & Roberson, Q. (2003). Why managers should care about fairness: The effects of aggregate justice perceptions on organizational outcomes. Journal of Applied Psychology, 88, 432–443. doi:10.1037/0021-9010.88.3.432.CrossRefPubMed

*Smith, K. G., Collins, C. J., & Clark, K. D. (2005). Existing knowledge, knowledge creation capability, and the rate of new product introduction in high-technology firms. Academy of Management Journal, 48, 346–357. doi:10.5465/AMJ.2005.16928421.CrossRef

*Sowinski, D. R., Fortmann, K. A., & Lezotte, D. V. (2008). Climate for service and the moderating effects of climate strength on customer satisfaction, voluntary turnover, and profitability. European Journal of Work and Organizational Psychology, 17, 73–88. doi:10.1080/13594320701473065.CrossRef

Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.). Washington, DC: American Council on Education.

*Stetzer, A., Morgeson, F., & Anderson, E. (1997). Organizational climate and ineffectiveness: Evidence from 25 outdoor workcrew divisions. Journal of Quality Management, 2, 251–265. doi:10.1016/S1084-8568(97)90006-7.CrossRef

Thoreson, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003). The affective underpinnings of job perceptions and attitudes: A meta-analytic review and integration. Psychological Bulletin, 129, 914–945. doi:10.1037/0033-2909.129.6.914.CrossRef

*Towler, A., Lezotte, D. V., & Burke, M. J. (2011). An examination of the service climate-firm performance chain: The role of customer retention. Human Resource Management, 50, 391–406. doi:10.1002/hrm.20422.CrossRef

*Wagner, S. H., Parker, C. P., & Christiansen, N. D. (2003). Employees that think and act like owners: Effects of ownerships beliefs and behaviors on organizational effectiveness. Personnel Psychology, 56, 847–871. doi:10.1111/j.1744-6570.2003.tb00242.x.CrossRef

Wallace, J. C., Edwards, B. D., Paul, J., Burke, M. J., Christian, M., & Eissa, G. (2016). Change the referent? A meta-analytic investigation of direct and referent-shift consensus models for organizational climate. Journal of Management, 42, 838–861.CrossRef

*Wallace, C., & Chen, G. (2006). A multilevel integration of personality, climate, self-regulation, and performance. Personnel Psychology, 59, 529–557. doi:10.1111/j.1744-6570.2006.00046.x.CrossRef

Whitman, D. S., Caleo, S., Carpenter, N. C., Horner, M. T., & Bernerth, J. B. (2012). Fairness at the collective level: A meta-analytic examination of the consequences and boundary conditions of organizational justice climate. Journal of Applied Psychology, 97(4), 776–791. doi:10.1037/a0028021.CrossRefPubMed

Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors, and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63(1), 41–81. doi:10.1111/j.1744-6570.2009.01162.x.CrossRef

*Zohar, D., & Luria, G. (2005). A multilevel model of safety climate: Cross level relationships between organization and group-level climates. Journal of Applied Psychology, 90, 616–628. doi:10.1037/0021-9010.90.4.616.CrossRefPubMed

Titel: Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients
verfasst von: Maura I. Burke
Ronald S. Landis
Michael J. Burke
Publikationsdatum: 26.07.2016
Verlag: Springer US
Erschienen in: Journal of Business and Psychology / Ausgabe 6/2017
Print ISSN: 0889-3268
Elektronische ISSN: 1573-353X
DOI: https://doi.org/10.1007/s10869-016-9464-7

Springer Professional

Abstract

Purpose

Design/Methodology/Approach

Findings

Implications

Originality/Value

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 6/2017

How Far Can Support Go? Supported Supervisors’ Performance and Subordinate Dedication

Reflection Toward Creativity: Team Reflexivity as a Linking Mechanism Between Team Goal Orientation and Team Creative Performance

The Creation of the Workplace Social Courage Scale (WSCS): An Investigation of Internal Consistency, Psychometric Properties, Validity, and Utility

Impression Management Use in Resumes and Cover Letters

Training Leader Emotion Regulation and Leadership Effectiveness

The Relationship of Coworker Incivility to Job Performance and the Moderating Role of Self-Efficacy and Compassion at Work: The Job Demands-Resources (JD-R) Approach