Estimating the robustness of questionnaire results: lessons from a mixed-mode survey of expectations for tele-working and road-based business travel

The robustness of questionnaire results to various forms of bias are explored in the context of a dual-mode (web and hardcopy) survey of employers’ anticipations of levels of employee commuting and business travel activity under a range of future ICT scenarios. The questionnaire incorporated several innovative features which, together with the dual-mode format, allowed an unusually wide range of analyses. For example: the robustness of respondents’ opinions was tested by examining the effect of incorporating alternative versions of a briefing text, one being very positive and one very negative, about the role of ICT; instrument bias was identified via detailed comparison of the results from the two versions of the questionnaire; and the impact of exogenous factors which are often ignored or taken as constant was assessed via special supplementary questions. Analysis showed that the robustness of opinions and expectations varied and was influenced by respondent characteristics, and that results from the two versions of the questionnaire differed significantly. It is concluded that opinions and expectations are less robust, and questionnaire results are more subject to bias and myopic interpretation, than is generally recognised and that web-based surveys seem particularly vulnerable to sampling bias. Methods are suggested for measuring robustness, for reducing bias and for validating and contextualising results. The use of contrasting briefing texts is recommended as a means of establishing the robustness of opinions and expectations while supplementary questions are recommended for validating and contextualising SP and SE exercises.