Effect of a first-time interviewer on cognitive interview quality

Cognitive interviews are a useful tool for questionnaire pretesting. However, detailed information on the proper way of performing them is scarce–researchers use this method without agreement on applicable techniques or specific rules on their realization. Additionally, users of cognitive interviewing techniques only seldom explain the pretesting results in their research, leaving it to readers to trust their outcomes. Follow-up observations on the success of the interviews and the overall experience are rarely, if ever, presented. We were interested in the learnability of the techniques when performed by non-experienced, newly instructed interviewers; moreover, we wanted to understand the most common problems they had experienced. During a five-year period, 120 methodology students performed 612 cognitive interviews and analyzed 17 different survey questionnaires. The precise documentation of their assignment served as a detailed database of qualitative and quantitative information on their experiences. The results present the first-time interviewers’ ability to accurately perform and analyze a cognitive interview. We show the most common mistakes and issues in all stages of the interviewing process, the influence of different interviewers’ and respondents’ characteristics, and the effect of the technique on the interview’s success.