The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

Factorial survey experiments (FSEs) have been used in the social sciences to study human judgments since the 1970s (Rossi 1979; Wallander 2009) and, in recent years, have become an established method to study how employers select job applicants (McDonald 2019). In a typical set-up, respondents are asked to evaluate fictitious descriptions of applicants, called “vignettes.” Researchers can study how employers interpret and value specific applicant characteristics during the hiring process through the experimental manipulation of information on applicants. Analyzing employers’ hiring preferences is crucial to our understanding of how inequalities arise at the individual and job level (Bills et al. 2017). Previous studies have employed FSEs to analyze, for example, gender discrimination (Kübler et al. 2018; Oesch et al. 2017), ethnic discrimination (Auer et al. 2019; Baert and De Pauw 2014), the value of foreign qualifications (Damelang et al. 2019; Protsch and Solga 2017), and the signaling value of educational credentials (Di Stasio 2014; Di Stasio and Van De Werfhorst 2016) and unemployment experience (Van Belle et al. 2018; Shi et al. 2018).

However, FSEs have been criticized for measuring the hiring intentions of employers, rather than their real behavior (e.g., Pager and Quillian 2005). Consequently, field experiments, such as correspondence or audit studies in which fictitious applications are sent to real jobs (e.g., Pedulla 2018; Protsch and Solga 2014), are often seen as better suited to detecting inequalities in the hiring process.¹

This criticism applies particularly to FSEs, in which respondents are put in a situation that may be completely unknown to them in real life. This would be the case, for example, if respondents are students (e.g., Baert and De Pauw 2014) or only partly familiar with the recruitment process for a specific job type (e.g., Van Belle et al. 2018; Damelang and Abraham 2016). However, FSEs can also be based on a careful selection of study participants, so that the pool of respondents closely matches the target population (i.e., individuals with recruiting experience for the occupation of interest and familiar with the situations described in the FSE). Adopting this strategy may increase external validity (Hainmueller et al. 2015).

Furthermore, FSEs have several advantages over field experiments, which make them an attractive option for studying hiring processes (see also, McDonald 2019). First, unlike field experiments, FSEs may capture the multidimensionality of hiring decisions by considering several applicant characteristics simultaneously instead of focusing on only one or two features. Moreover, the survey part of FSEs allows the researcher to gather rich information about recruiters, companies, and jobs; this enables an in-depth analysis of the mechanisms underlying recruiters’ decision-making processes, an analytical feature typically unavailable in field experiments. Finally, FSEs are easily compatible with ethical research practices, as the respondents (i.e., recruiters) give informed consent to participate in the survey, whereas in field experiments, real employers are led to believe that they are dealing with real applications.²

Resolving the FSE versus field experiment debate is beyond the scope of this article. Given that prior research suggests that FSEs are useful, some of the justified criticisms about their use should be addressed by improving the design of the FSEs applied to study hiring intentions. A research group from the NEGOTIATE project proposed basing FSEs on samples of current real-world vacancies and the recruiters responsible for filling them (NEGOTIATE 2020).³ This is in contrast to most previous FSEs on recruiting, which relied on rather abstract descriptions of hypothetical vacancies that may have lacked psychological realism (e.g., Van Belle et al. 2018; Damelang et al. 2019; Di Stasio 2014). “Psychological realism [emphasis in original] is the extent to which the psychological processes that occur in an experiment are the same as the psychological processes that occur in everyday life” (Brewer and Crano 2014, p. 21). This is an important factor that should be considered when designing FSEs, to enhance the external validity (Auspurg and Hinz 2015). Conducting FSEs with recruiters who are currently in charge of filling a real-world vacancy might increase psychological realism and, hence, contribute to the internal and external validity of the results (NEGOTIATE 2020). However, collecting large sets of job advertisements and employer details is costly and time-consuming. Furthermore, given that respondents are still presented with a fictitious situation that entails no real-life consequences, the question arises whether it really makes a difference for the measured hiring intentions of recruiters whether vignette ratings refer to a real hiring problem or a hypothetical one. If the FSE is based on hypothetical vacancies, the employers might have trouble putting themselves in the actual hiring situation, particularly when the respondents are not familiar with recruiting for the specific job type studied. Hence, the internal and external validity might be reduced. This is presumably different if the FSE is conducted with recruiters who are currently responsible for filling a real-world vacancy on which the experiment is based. Only the applicant profiles are fictitious in this case, which might increase internal and external validity. However, the question remains whether this is enough to trigger more valid judgments compared to a completely hypothetical situation.

The main objective of this study is to help answer this question. While the advantages and disadvantages of survey experiments in general have been discussed (e.g., Sniderman 2018), more research is needed on the designs of FSEs to study hiring intentions. We conducted an FSE in Luxembourg, where one group of recruiters rated hypothetical applicants for a real vacancy for which they were personally responsible in real life (i.e., for which they know all the requirements and characteristics) and another group of recruiters rated hypothetical applicants for a hypothetical vacancy (i.e., without information on job characteristics). Since random assignment of recruiters to a condition with real or hypothetical vacancies (i.e., a split ballot experiment) was impossible due to data limitations, we compared two different samples of recruiters. We manipulated three applicant characteristics: migrant background, gender, and unemployment. We asked whether the effects of these characteristics on observed hiring intentions differed if the rating task corresponded to a real instead of a hypothetical hiring problem and whether these effects were closer to certain benchmarks when real vacancies were used. “Behavioral benchmarks” (see Hainmueller et al. 2015) corresponding to real-life hiring behavior were not available for our study context, which is why we derived “theoretical” benchmarks from previous research. Given the growing number of FSEs in employer studies and the potential implications for the validity of those studies’ results, answering this question is relevant for both methodological and practical reasons. However, to the best of our knowledge, this question has not been addressed empirically.

After a literature review in Sect. 2, we discuss our expectations about the impact of the FSE design on recruiters’ stated hiring intentions in Sect. 3. In Sect. 4, we describe our research design. Our results are presented in Sect. 5 Section 6 contains our conclusions.

Table 1 provides an overview of previous FSEs used to study recruiters’ hiring intentions.⁴ All listed studies ask respondents to rate either the hiring chances of applicants or the likelihood of inviting applicants to the next selection stage (e.g., interview). Moreover, in all cases, the presented applicant profiles (i.e., vignettes) are fictitious. These studies concern how recruiters interpret and value certain applicant characteristics, such as educational background, age, or nationality.

The last column of Table 1 shows the type of jobs presented to respondents. To the best of our knowledge, so far only in NEGOTIATE (2020) have respondents been asked about a real vacancy in their company that they are responsible for filling.⁵ Scholars interested in measuring hiring intentions by means of FSEs clearly prefer using hypothetical vacancies. However, the presentation of hypothetical jobs to respondents varies in certain ways. For example, some studies give specific information about the job, such as characteristics required for the position (e.g., Baert and De Pauw 2014; De Wolf and Van Der Velden 2001); other studies are more general and provide no details about the job (e.g., Oesch et al. 2017). Furthermore, some rating tasks do not refer to a job per se but to the organization as a whole (e.g., Karpinska et al. 2011).

The pool of respondents used in the reviewed studies also differs noticeably. The respondents in most studies are managers or human resource professionals, who have at least some recruitment experience (see the 5th column in Table 1). Yet, they are not always experts in the specific occupations under investigation or in the relevant recruitment procedures (e.g., Damelang and Abraham 2016). In a few cases, the sample consists fully or partly of students (Baert and De Pauw 2014; Karpinska et al. 2011). Such a mismatch between the target population (i.e., real recruiters with job-specific knowledge) and the analytic sample could affect the internal and external validity of the results, as discrepancies between measured intentions and real behavior are more likely if respondents are not familiar with the rating task in real life (see Todesco 2017, p. 17). Hence, FSEs based on hypothetical vacancies can enhance the validity of the results by carefully selecting respondents with relevant recruiting experience (ideally for the types of jobs under investigation) and designing hypothetical vacancies that are familiar to them. Nevertheless, this issue can be easily addressed if the sample consists of real-world vacancies, given that, in this case, the job described in the hiring situation is concrete and real for the respondents. The respondents (i.e., real recruiters) are asked about a currently vacant position in their company, for which they are personally responsible. This means that they know all the relevant characteristics of this vacancy and do not have to make any assumptions when rating vignettes (see Sect. 3). This is different for respondents who may have relevant recruiting experience but are presented with a rather vague description of a hypothetical vacancy.

Further differences among FSEs used to measure hiring intentions involve the type of samples used. Respondents were either drawn from a representative population or employer surveys or were sampled specifically for the respective study context (see column 4 in Table 1). For example, Liechti et al. (2017) sampled respondents based on the membership register of a Swiss hotel employer association, whereas other studies used a (random) sample of firms to find suitable study participants (e.g., Van Beek et al. 1997; Di Stasio 2014). Similar to the NEGOTIATE research group, Petzold (2017) sampled real job advertisements through which recruiters were contacted. As far as we can judge, however, Petzold (2017), in contrast to NEGOTIATE (2020), presented respondents with a hypothetical job description.

Some of these differences may be due to the different study contexts and goals, but they also indicate an evolving field that has not yet converged upon shared standards. Whereas the vast majority of the reviewed studies arrive at conclusions in line with the respective theory, common guidelines will increase their comparability.

Our contribution focuses on the applicant’s migrant background, gender, and unemployment experience to examine whether the FSE design affects recruiters’ hiring intentions. Hiring-related inequalities based on these characteristics have been of particular interest to scholars in labor market research. In the following, we develop the argument that the effects of these characteristics on recruiters’ hiring intentions are closer to what we would observe in reality when real vacancies are used. To test this hypothesis, behavioral benchmarks corresponding to the true effects of these characteristics on hiring decisions would be ideal (see Hainmueller et al. 2015), but these were not available for our study context. Therefore, we had to rely on the extensive empirical evidence on the effects of our three applicant characteristics on recruiters’ hiring decisions to find plausible “theoretical” benchmarks. These will be briefly discussed below, before we turn to our expectations about the impact of using real vacancies in FSEs on recruiters’ hiring intentions.

In line with well-established discrimination theories, hiring discrimination toward foreigners and ethnic minorities has been documented through numerous field (e.g., Zschirnt and Ruedin 2016; Quillian et al. 2019) and survey experiments (e.g., Auer et al. 2019). Therefore, we assume a negative effect of a foreign background on recruiters’ hiring intentions as the benchmark for the effect of migrant background for our study context. Moreover, experimental studies suggest an advantage for women in female-dominated occupations and a disadvantage for women in male-dominated occupations (e.g., Birkelund et al. 2019; Koch et al. 2015; Fernandez and Mors 2008). Based on this evidence, we assume two benchmarks for the effect of gender for our study context: a negative (positive) effect of being female in male-dominated (female-dominated) occupations. However, the empirical evidence on the effect of gender on recruitment decisions is somewhat mixed, and these benchmarks might therefore be less reliable. For example, some studies contest the notion of hiring discrimination towards women in male-dominated occupations (e.g., Petersen and Togstad 2006; Carlsson 2011). Finally, with only a few exceptions (e.g., Nunley et al. 2017), several experimental studies have shown that employers are more reluctant to hire applicants who have been unemployed (e.g., Van Belle et al. 2018; Eriksson and Rooth 2014; Kroft et al. 2013; Oberholzer-Gee 2008). Following this literature, we assume a negative effect of unemployment on recruiters’ hiring intentions as our benchmark.

We explore why and how these effects might differ when the rating task refers to a real hiring problem as opposed to a hypothetical one. First, the hypothetical job descriptions and vignettes presented in prior FSEs likely differ from real-life hiring situations, which might lead to a “hypothetical bias” (Ajzen et al. 2004), meaning that respondents answer differently than they would in reality. However, the psychological realism of the rating task increases decisively when the FSE is based on real vacancies for which the respondents are personally responsible in real life. In this case, the vacancies are familiar to the respondents, and they can draw on the same information (e.g., regarding requirements) they would normally do to form hiring decisions for this particular job type. Although hypothetical vacancies might allow researchers to control for possible differences in job requirements, it is difficult to determine whether the information provided in fictitious job descriptions will sufficiently chime with the psychological realism hold by respondents. A careful design, such as asking recruiters about typical vacancies in their company (e.g., Van Beek et al. 1997; Di Stasio 2014), could alleviate the potential problem of low psychological realism. Nevertheless, respondents must rely on assumptions about requirements and characteristics of the respective hypothetical vacancy on which the experiment is based, which may differ from real vacancies. The latter is all the more likely the less familiar the respondents are with typical vacancies in the respective occupation studied and the less specific the descriptions of these vacancies are. In contrast, while vignettes only represent simplified applicant profiles (i.e., they contain less information than real application documents), they usually provide realistic information about applicants according to the researchers’ interest (if carefully designed).⁶ This holds regardless of whether real or hypothetical vacancies are used. Yet, respondents who are presented with a real vacancy will have less difficulty deciding whether the provided information corresponds to reality and matches the respective job characteristics. As a corollary, the information respondents consider when rating the vignettes might be closer to real-life decisions made throughout the hiring process. Therefore, the main effects of gender, migration background, and unemployment should be closer to our derived benchmarks if real vacancies are used.

Second, social desirability is a well-known problem in survey research that might bias results. Although the multidimensional design of FSEs should mitigate the risk of normative influences on vignette ratings (Auspurg et al. 2015), the possibility of social desirability bias cannot be fully excluded. Socially biased results would imply less discrimination by gender or migration background and less negative unemployment effects than theoretically expected. Recruiters might be more cautious with their answers in both FSE versions, as they cannot be sure about the real reason for the experiment; however, they might feel more confident about their vignette ratings when the FSE is based on real vacancies. This is because, as we have argued, real vacancies provide psychological realism to the rating task and recruiters may find it easier to justify their answers if needed. For similar reasons, the use of real vacancies might also reduce the risk of other response biases, such as acquiescence (Krosnick et al. 2014; Schwarz 1999), as respondents might be generally less susceptible to external influences if they are confronted with a real-life hiring problem. Consequently, in line with our considerations above, the effects of applicant characteristics should be closer to our benchmarks when real instead of hypothetical vacancies are used. However, the effects of applicant characteristics might not be affected in the same way. Reluctance to hire based on unemployment could be more easily justified with productivity-related characteristics (e.g., skill loss) than reluctance based on characteristics such as gender or migration background, where social desirability is more likely an issue.

Third, FSEs have been criticized for presenting simplified hiring situations that cannot possibly convey the urgency and pressures surrounding actual hiring (e.g., difficulty finding suitable candidates or a company’s need to fill a position promptly). These aspects are more likely to be considered if respondents are confronted with a real hiring problem. Prior research suggests that gender, unemployment, or migration background might be less relevant in such a case (e.g., Baert et al. 2015). Unfortunately, we cannot analyze aspects of urgency in detail due to data limitations (see Sect. 4).

Against this background, we expect differences in the effect of applicant characteristics on recruiters’ hiring intentions based on the FSE’s design. More specifically, in line with our theory, we expect the effects of gender, migrant background and unemployment on vignette ratings to be closer to the defined benchmarks when real vacancies are used as compared to hypothetical vacancies (Hypothesis 1). Although in both cases, respondents rate hypothetical applicants without real-life consequences, we argue that the perceived realism associated with the rating task is markedly higher in the former setting. Also, the effect of unemployment might be generally less affected by social desirability than the effects of gender and migration background are. Hence, we expect the difference in the effects of gender and migrant background between the FSE designs (see Hypothesis 1) to be greater than the same difference regarding the effect of unemployment (Hypothesis 2). The hypotheses will be tested for each applicant characteristic separately.

We conducted an FSE in Luxembourg in 2018 and 2019.⁷ To enable comparability with NEGOTIATE (2020), we built on that design using the same pictorial representation of CV and answer scale.⁸ Respondents were asked to rate hypothetical descriptions of applicants (vignettes) that varied systematically on a number of different characteristics (dimensions). The rating task either referred to a real vacancy in their company or to a similar but hypothetical job.

The relatively small Luxembourgish labor market did not provide enough real vacancies to conduct a split ballot experiment, where we could have randomly assigned respondents to a real vacancy or a hypothetical job description in the FSE.⁹ Therefore, we employed a two-step approach to gather our data. First, we collected real vacancies and recruiter contact information published on different online job portals and company websites in Luxembourg. Second, we sampled recruiters from publicly available lists of companies and businesses associated with the same occupational fields. Respondents sampled based on the former approach rated hypothetical applicants in the FSE based on the real vacancies they were responsible for filling. Respondents sampled based on the latter approach rated applicants for a hypothetical but similar type of job. In the present analysis, we focus on jobs in catering and in information technology (IT).¹⁰ Both groups of respondents have at least some recruitment experience in the respective occupations studied (see Sects. 4.2 and 4.3), but only respondents sampled based on real vacancies are presented with a concrete, realistic hiring problem in the FSE. This enables us to test whether the use of real vacancies triggers different responses while avoiding distortion in our results from differences in the recruitment experience between the two samples (and thus fundamental differences in perceived realism). In the following sections, we will refer to the “real vacancy” (RV) design and “hypothetical vacancy” (HV) design when discussing the two versions of the FSE. However, the reader should keep in mind that we are essentially comparing two different samples.

In this study, we conducted an FSE for two occupational fields (catering and IT) in Luxembourg to examine whether the effects of applicant characteristics on observed hiring intentions differ when the vignette ratings refer to a real instead of a hypothetical vacancy. We focused on three applicant characteristics in our analyses: migration background, gender, and unemployment.

Estimating multilevel regressions, our results regarding the main effects of applicant characteristics largely correspond to our benchmarks. However, we observed some differences based on the occupational field. Regarding the migration background, applicants with a foreign background face disadvantages in catering, which holds for both FSE versions. In the RV version, the effect was slightly less negative than in the HV version. In contrast, migration background had no significant effect on vignette ratings in IT, although the respective effect tended toward different directions depending on the FSE design. The effect of gender (1=female) tended to be positive in both FSE versions in catering but only in the HV version in IT. In the RV version in IT, the effect of gender was negative. Women might have an advantage in catering because service jobs are typically considered to be a “female” job (e.g., Booth and Leigh 2010). A female disadvantage in IT is, in turn, in line with prior research suggesting that women are discriminated in STEM (science, technology, engineering, math) occupations (e.g., Kübler et al. 2018). However, the effect of gender was not significant in IT in our study. Finally, we found very similar results regarding unemployment for both occupations: in both cases, unemployment significantly reduced hiring intentions. The respective effect rarely differed between the FSE designs in IT but was slightly less negative in the RV version in catering. The differing patterns in the observed effects between occupations might be explained by general structural differences between the two sectors.

Overall, none of the observed differences in the effects between the FSE versions were statistically significant. A more fine-grained analysis of single categories of migration background revealed that French border workers were particularly disadvantaged in IT in the RV version. In the HV version in IT, the effect of being a French border worker tended to be positive but was not significant. The observed difference was statistically significant at the 10% level. This finding lends some support to the use of real-world vacancies, as our benchmark suggested a negative effect of being a non-native person on recruiters’ hiring intentions. Excluding this finding, our results provide little evidence for differences in the effect of applicant characteristics on vignette ratings between RV and HV FSE designs.

We observed in this study and the pilot study that response rates in the RV design were about twice those in the HV design of the FSE. If response rates are of special concern, efficient sampling of real vacancies might be an attractive option. Most importantly, higher involvement of recruiters in the FSE due to real vacancies probably also contributes to higher quality of answers and higher internal validity of the results.

This study has some limitations, which make an overall conclusion about the impact of the FSE design on hiring intentions difficult. Our findings must therefore be interpreted with caution. First, our small sample of real vacancies made it impossible to employ a true split ballot experiment. A random assignment of respondents to one of the two FSE designs would have provided a more robust estimate of the effect of using real vacancies. Instead, we compared two different samples that slightly differed in the composition of the respondents: (i) actual recruiters responsible for filling a real vacancy in their company and (ii) a mix of human resource professionals, managers, and directors. Hence, the comparability of the two samples is limited. In our regression analyses, we have used weights to balance both FSE samples for key respondent characteristics. However, using weights does not change our main findings compared to an analysis without weights. Additional analyses, which we present in the online Appendix, further revealed that the randomization processes within our experiment were successful. We found correlations very close to zero and not statistically significant between vignette variables and observed respondent characteristics (see Table A3 and A4). We also estimated fixed effects regressions with vignette variables as predictors to test the influence of unobserved respondent characteristics (Auspurg and Hinz 2015). The correlation (\(u_j\), X) between the error term and control variables can be taken as indicator for omitted respondent characteristics that influence vignette ratings. In both occupations, these correlations were close to zero (see Table A5). Second, to reliably test whether using real vacancies in FSEs leads to more valid vignette ratings than using hypothetical vacancies, behavioral benchmarks reflecting the true effects of our applicant characteristics on hiring decisions would have been desirable; however, these were not available for Luxembourg. Hence, we compared our results to “theoretical” benchmarks derived from previous experimental research from other contexts (see Sect. 3). Third, our sample size was rather small. Some of our null findings might be due to limited statistical power because of the low number of observations.

Moreover, we were unable to go beyond a broad comparison of two occupations (catering and IT specialists) and consider potential contextual moderators more in detail. The job requirements as well as structural characteristics (e.g., unemployment rates and labor market tightness) typically differ between labor market sectors. These factors may have an impact on how employers interpret and act upon the information provided in vignettes. While this limits the generalizability of our findings concerning the main effects of applicant characteristics on recruiters’ hiring intentions, it should not affect the generalizability of our methodological effects. Nevertheless, as stated in Sect. 3, differences in vignette ratings when real vacancies as opposed to hypothetical ones are used might be particularly pronounced if the urgency of finding suitable candidates is high. This might be the case, for example, if recruitment is perceived to be difficult. Most respondents in our sample generally found it difficult to recruit suitable workers,²² which is why a comparative analysis concerning the perceived difficulty to find applicants was not possible due to low variance.

Finally, the idiosyncrasies of Luxembourg limit the comparability of our results with other studies. For example, the Luxembourgish labor market is unique in the multi-cultural composition of its workforce. As aforementioned, different outcomes particularly regarding the main effects of applicant characteristics on recruiters’ hiring intentions may be expected in other contexts.

The question of whether we should use real or hypothetical vacancies when studying hiring prompts more general considerations about sampling and inference. Researchers will have to assess whether they wish to make statements about mechanisms at the level of companies, occupations, vacancies, or recruiters or a combination of these. For example, if the main interest is to investigate potential discrimination, it will be beneficial to sample randomly from a pool of actually available vacancies, as they may differ systematically from filled positions in a given occupation. If the primary aim is to advance our general understanding of recruiter behavior in a given sector, on the other hand, presenting hypothetical vacancies to a random selection of recruiters from this sector might be adequate.

This study draws attention to potential issues surrounding the routine use of hypothetical vacancies in FSEs, an area that has been widely neglected in previous studies on recruiters’ hiring intentions. Our results suggest that using hypothetical vacancies might be, within the limits of FSEs, a valid approach. Note, however, that the effect of using real vacancies might have been more pronounced if the differences between our two samples would have been larger (e.g., recruiters with only general recruiting experience vs. recruiters with job-specific recruiting knowledge). However, identifying the effect of using real vacancies would have been more difficult with such an approach, as both the type of sample and the type of vacancy potentially affect the perceived realism of the rating task.

In any case, the potential effect of using real-world vacancies in FSEs on response behavior deserves further scrutiny. This seems particularly important given the potential methodological and practical implications for the increasing number of FSEs in employer studies. Substantial and theoretically plausible differences between FSE designs would further encourage the use of real vacancies as the internal and external validity is likely increased. Besides aspects that could not be analyzed in this study (e.g., the role of recruitment difficulties), the present analyses could be extended comparing the survey responses from vignette ratings referring to real vacancies with behavioral data to test the internal and external validity of the results obtained from both types of FSE (i.e., RV versus HV design). In doing so, researchers should ensure comparability in experimental designs and sample composition between studies (Petzold and Wolbring 2019).

A better understanding of the implications of the differences in FSE designs between studies is surely needed to establish best practices in employer studies. Prior research on general methodological questions related to the design of FSEs (e.g., Auspurg et al. 2019; Auspurg and Jäckle 2017; Sauer et al. 2011; Shamon et al. 2019) continues to be highly relevant for the application of FSEs to study employer preferences, as it provides common guidelines for designing vignettes and answer scales. More studies on methodological aspects of FSEs are required, however, to advance our understanding of their possibilities and limits in research on hiring. We hope that our comparison of designs using real and hypothetical vacancies contributes to this emerging strand of methodological inquiry.