Skip to main content
main-content
Top

Hint

Swipe to navigate through the articles of this issue

Published in: Journal of Business and Psychology 4/2021

05-06-2020 | Original Paper

Performance on Video-Based Situational Judgment Test Items: Simulated Interracial Interactions

Authors: Juliya Golubovich, Ann Marie Ryan

Published in: Journal of Business and Psychology | Issue 4/2021

Login to get access
share
SHARE

Abstract

Individuals are known to categorize others into social groups based on cues like race and gender and to experience relative discomfort when interacting with “outgroup” members. Two experimental studies were used to examine whether actor demographic cues in situational judgment assessment items completed by test takers in a simulated employee selection context may lead to differences in their performance and reactions to the hiring organization. In both studies, test takers assumed the perspective of actors shown in video-based scenarios and indicated how they would respond to interaction partners (IPs) to whom they were racially similar or dissimilar. In Study 1, a given test taker responded to IPs of a constant gender; in Study 2, IPs’ gender varied across scenarios within each condition. In Study 1, Black test takers spent more time and scored better on two of the four scenarios when responding to racially similar IPs. These effects were not found in Study 2, but demographic cues showed new interactive effects on performance and reactions. We discuss the implications of different findings across the two studies.
Footnotes
1
Campion et al. (2014) reviewed SJT research and correlated SJT attributes with their coefficient alpha values. Alpha values were lower when video versus written SJT formats were used (r = − .28, p < .05).
 
2
Brief interactions with or observations of strangers differ from the more sustained interactions over time examined in the context of cross-group friendships (e.g., West and Dovidio 2013). Friendship development involves more complex and dynamic processes where interaction partners influence each other and evidence change in personal attitudes and behavior over time (West and Dovidio 2013). For example, as individuals get to know outgroup members, they can find out individuating information that may be inconsistent with their held stereotypes (Rothbart and John 1985). Thus, we bound our literature review and theoretical rationale to contexts of interracial interactions between individuals who are new interaction partners or who have less close relationships and not to discussions of long-standing, well-developed close relationships.
 
3
As hypothesis testing was based on White, Asian, and Black actors, additional participants who were not from those demographic groups (N = 21) were excluded from analyses.
 
4
Research on avatar gender has shown that as many as 79% of those in multiplayer online games have used avatars of an opposite gender and 30% do so on a regular basis (Hussain and Griffiths 2008; Yee & Ducheneaut, 2011, as cited by Martey et al. 2014). Findings fairly consistently show that when assuming an avatar of a different gender people do not tend to behave differently (i.e., movement within a game), but when men take on a female role, they tend to adopt stereotypic language patterns in chat functions (i.e., use more emotional language, use more exclamation points; Yee and Bailenson 2007; Martey et al. 2014). As our response options were not gendered in language patterns and individuals were responding to a multiple choice item, this research would suggest that gender assumed should not have an effect. Men and women in the current research generally did not perform differently on the SJT scenarios.
 
5
Two measures of reactions to the SJT scenarios—perceived opportunity to perform and job relatedness—were also included in the current studies, but there were no differences in these measures across conditions in either Study 1 or Study 2. In the interests of brevity, these measures are not discussed. Ratings of interaction partner attractiveness were also collected. As these ratings were generally uncorrelated with the outcomes of interest and controlling for these ratings in testing the hypotheses and research questions in studies 1 and 2 did not substantively change our results, these measures are likewise not discussed.
 
6
The pattern of significant results did not change when rerunning analyses without excluding multivariate outliers.
 
7
We crossed participant race and IP race to check if results differed depending on the race of the IP and found that for the conflict situation, Black test takers performed better when interacting with a Black IP than when interacting with the White IP. For the helping situation, Black test takers performed better when interacting with a Black IP than when interacting with either the White IP or the Asian IP. However, sample sizes are small for cells crossing Black participants with Asian and White IPs (Ns = 8–9).
 
8
When crossing test taker race and IP race, Black test takers spent significantly more time responding to Black IPs than to Asian IPs for all four scenarios. Response time differences for Black IPs relative to White IPs were statistically significant for conflict situation 2 and helping situation 2. Again, sample sizes are small for cells crossing Black participants with Asian and White IPs (Ns = 8–9).
 
9
Additional exploratory analyses examined whether test taker-IP gender similarity affected SJT item scores, item response times, or perceived fit with the organization and whether test taker gender interacted with test taker race, test taker-IP racial similarity, or level of diversity represented in the SJT videos. There were no statistically significant effects in Study 1.
 
10
The pattern of significant results did not change when rerunning analyses without excluding multivariate outliers.
 
11
There were also no statistically significant differences in performance when crossing test taker race and IP race.
 
12
We also crossed test taker race and IP race and found that Black test takers spent more time responding to the two conflict situations when interacting with Asian IPs. For conflict situation 1, the difference was statistically significant relative to White IPs; for conflict situation 2, the difference was statistically significant relative to both White and Black IPs.
 
13
Additional exploratory analyses examined whether test taker-IP gender similarity affected SJT item scores, item response times, or perceived fit with the organization and whether test taker gender interacted with test taker race, test taker-IP racial similarity, or level of diversity represented in the SJT videos. Like in Study 1, there were no statistically significant effects.
 
14
For interactions involving six groups (e.g., race similarity × test taker race), a sample size of approximately 35 per group is recommended to detect medium effect sizes at a .05 significance level with a power of .80 (Cohen 1992). After data cleaning, our sample sizes of minority test takers ranged from 11 to 18 in Study 1 and 15 to 36 in Study 2.
 
Literature
go back to reference Bauer, T. N., & Truxillo, D. M. (2006). Applicant reactions to situational judgment tests: Research and related practical issues. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 233–251). Mahwah, NJ: Erlbaum. Bauer, T. N., & Truxillo, D. M. (2006). Applicant reactions to situational judgment tests: Research and related practical issues. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 233–251). Mahwah, NJ: Erlbaum.
go back to reference Bodenhausen, G. V., & Macrae, C. N. (1998). Stereotype activation and inhibition. In R. S. Wyer Jr. (Ed.), Stereotype activation and inhibition: Advances in social cognition (pp. 1–52). Hillsdale, NJ: Erlbaum. Bodenhausen, G. V., & Macrae, C. N. (1998). Stereotype activation and inhibition. In R. S. Wyer Jr. (Ed.), Stereotype activation and inhibition: Advances in social cognition (pp. 1–52). Hillsdale, NJ: Erlbaum.
go back to reference Boyce, A.S, Corbet, CE & Adler, S (2013). Simulations in the selection context: Considerations, challenges, and opportunities. In M. Fetzer & K. Tuzinski (Eds.). Simulations for personnel selection (pp17–41). New York, NY: Springer. Boyce, A.S, Corbet, CE & Adler, S (2013). Simulations in the selection context: Considerations, challenges, and opportunities. In M. Fetzer & K. Tuzinski (Eds.). Simulations for personnel selection (pp17–41). New York, NY: Springer.
go back to reference Carli, L. L. (1989). Gender differences in interaction style and influence. Journal of Personality and Social Psychology, 56(4), 565–576. CrossRef Carli, L. L. (1989). Gender differences in interaction style and influence. Journal of Personality and Social Psychology, 56(4), 565–576. CrossRef
go back to reference Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. CrossRef Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. CrossRef
go back to reference Crenshaw, K. (1989/1993). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory and antiracist policies. In D. K. Weisbert (Ed.), Feminist legal theory: Foundations (pp. 383–395). Philadelphia, PA: Temple University Press. (Original work published 1989). Crenshaw, K. (1989/1993). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory and antiracist policies. In D. K. Weisbert (Ed.), Feminist legal theory: Foundations (pp. 383–395). Philadelphia, PA: Temple University Press. (Original work published 1989).
go back to reference de Meijer, L. A. L., Born, M. P., van Zielst, J., & van der Molen, H. T. (2010). Construct-driven development of a video-based situational judgment test for integrity: A study in a multi-ethnic police setting. European Psychologist, 15, 229–236. CrossRef de Meijer, L. A. L., Born, M. P., van Zielst, J., & van der Molen, H. T. (2010). Construct-driven development of a video-based situational judgment test for integrity: A study in a multi-ethnic police setting. European Psychologist, 15, 229–236. CrossRef
go back to reference Dovidio, J. F., & Gaertner, S. L. (2004). Aversive racism. In M. Zanna (Ed.), Advances in experimental social psychology (Vol. 36, pp. 1–52). San Diego, CA: Elsevier Academic Press. Dovidio, J. F., & Gaertner, S. L. (2004). Aversive racism. In M. Zanna (Ed.), Advances in experimental social psychology (Vol. 36, pp. 1–52). San Diego, CA: Elsevier Academic Press.
go back to reference Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum. Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum.
go back to reference Hawkes, B., Cek, I., & Handler, C. (2018). The gamification of employee selection tools: An exploration of viability, utility, and future directions. In J. C. Scott, D. Bartram, & D. H. Reynolds (Eds.), Next generation technology-enhanced assessment: Global perspectives on occupational and workplace testing (pp. 288–316). Cambridge, UK: Cambridge University Press. Hawkes, B., Cek, I., & Handler, C. (2018). The gamification of employee selection tools: An exploration of viability, utility, and future directions. In J. C. Scott, D. Bartram, & D. H. Reynolds (Eds.), Next generation technology-enhanced assessment: Global perspectives on occupational and workplace testing (pp. 288–316). Cambridge, UK: Cambridge University Press.
go back to reference Lievens, F. (2013). Adjusting medical school admission: Assessing interpersonal skills using situational judgement tests. Medical Education, 47, 182–189. CrossRef Lievens, F. (2013). Adjusting medical school admission: Assessing interpersonal skills using situational judgement tests. Medical Education, 47, 182–189. CrossRef
go back to reference Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority–majority differences and validity. Journal of Applied Psychology, 104, 715–726. CrossRef Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority–majority differences and validity. Journal of Applied Psychology, 104, 715–726. CrossRef
go back to reference MacCann, C., Lievens, F., Libbrecht, N., & Roberts, R. D. (2016). Differences between multimedia and text-based assessments of emotion management: An exploration with the multimedia emotion management assessment (MEMA). Cognition and Emotion, 30, 1317–1331. CrossRef MacCann, C., Lievens, F., Libbrecht, N., & Roberts, R. D. (2016). Differences between multimedia and text-based assessments of emotion management: An exploration with the multimedia emotion management assessment (MEMA). Cognition and Emotion, 30, 1317–1331. CrossRef
go back to reference Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Science of India, 12, 49-55. Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Science of India, 12, 49-55.
go back to reference Murphy, M. C., & Taylor, V. J. (2012). The role of situational cues in signaling and maintaining stereotype threat. In M. Inzlicht & T. Schmader (Eds.), Stereotype threat: Theory, process, and application; stereotype threat: Theory, process, and application (pp. 17–33). New York, NY: Oxford University Press. Murphy, M. C., & Taylor, V. J. (2012). The role of situational cues in signaling and maintaining stereotype threat. In M. Inzlicht & T. Schmader (Eds.), Stereotype threat: Theory, process, and application; stereotype threat: Theory, process, and application (pp. 17–33). New York, NY: Oxford University Press.
go back to reference O’Connell, M., Lawrence, A., & Kinney, T. (2013). Show me you can do it: The use of interactive simulations in manufacturing settings. In M. Fetzer & K. Tuzinski (Eds.), Simulations for personnel selection (pp. 157–185). New York, NY: Springer. CrossRef O’Connell, M., Lawrence, A., & Kinney, T. (2013). Show me you can do it: The use of interactive simulations in manufacturing settings. In M. Fetzer & K. Tuzinski (Eds.), Simulations for personnel selection (pp. 157–185). New York, NY: Springer. CrossRef
go back to reference Ryan, A. M., Inceoglu, I., Bartram, D., Golubovich, J., Grand, J. A., Reeder, M., Derous, E., Nikolaou, I., & Yao, X. (2015). Trends in testing: Highlights of a global survey. In I. Nikolaou & J. K. Oostrom (Eds.), Employee recruitment, selection, and assessment: Contemporary issues for theory and practice (pp. 136–153). New York, NY: Psychology Press. Ryan, A. M., Inceoglu, I., Bartram, D., Golubovich, J., Grand, J. A., Reeder, M., Derous, E., Nikolaou, I., & Yao, X. (2015). Trends in testing: Highlights of a global survey. In I. Nikolaou & J. K. Oostrom (Eds.), Employee recruitment, selection, and assessment: Contemporary issues for theory and practice (pp. 136–153). New York, NY: Psychology Press.
go back to reference Tajfel, H., & Turner, J. C. (1979). An integrative theory of intergroup conflict. In W. G. Austin & S. Worchel (Eds.), The social psychology of intergroup relations (pp. 33–48). Monterey, CA: Brooks/Cole. Tajfel, H., & Turner, J. C. (1979). An integrative theory of intergroup conflict. In W. G. Austin & S. Worchel (Eds.), The social psychology of intergroup relations (pp. 33–48). Monterey, CA: Brooks/Cole.
go back to reference Tippins, N. T., & Adler, S. (2011). Technology-enhanced assessment of talent. San Francisco, CA: Jossey-Bass. CrossRef Tippins, N. T., & Adler, S. (2011). Technology-enhanced assessment of talent. San Francisco, CA: Jossey-Bass. CrossRef
go back to reference Tsui, A. S., Egan, T. D., & O’Reilly III, C. A. (1992). Being different: Relational demography and organizational attachment. Administrative Science Quarterly, 37, 549–579. CrossRef Tsui, A. S., Egan, T. D., & O’Reilly III, C. A. (1992). Being different: Relational demography and organizational attachment. Administrative Science Quarterly, 37, 549–579. CrossRef
go back to reference West, T. V., & Dovidio, J. F. (2013). Intergroup contact across time: Beyond initial contact. In G. Hodson & M. Hewstone (Eds.), Advances in intergroup contact (pp. 152–175). New York, NY: Psychology Press. West, T. V., & Dovidio, J. F. (2013). Intergroup contact across time: Beyond initial contact. In G. Hodson & M. Hewstone (Eds.), Advances in intergroup contact (pp. 152–175). New York, NY: Psychology Press.
Metadata
Title
Performance on Video-Based Situational Judgment Test Items: Simulated Interracial Interactions
Authors
Juliya Golubovich
Ann Marie Ryan
Publication date
05-06-2020
Publisher
Springer US
Published in
Journal of Business and Psychology / Issue 4/2021
Print ISSN: 0889-3268
Electronic ISSN: 1573-353X
DOI
https://doi.org/10.1007/s10869-020-09697-1

Other articles of this Issue 4/2021

Journal of Business and Psychology 4/2021 Go to the issue

Premium Partner