Response willingness in consecutive travel surveys: an investigation based on the National Household Travel Survey using a sample selection model

High-quality survey data provide the foundation for research and policymaking across many fields. While novel data sources are actively being examined for use in transport applications, both currently and for the foreseeable future traditional travel surveys will continue to play an irreplaceable role in providing critical data for use in travel demand modeling, regional planning, and policymaking. However, survey response rates are in continuous and significant decline, thus requiring increased efforts toward respondent recruitment. Further necessitating these increased efforts is the fact that low response rates and their accompanying nonresponse biases can threaten the validity of survey data, and thus contingent research findings (National Research Council 2013).

Survey teams have employed a range of efforts aimed at increasing response rates and improving survey data quality. Among the most common tools are the use of passive datasets such as GPS records (Bohte and Maat 2009) and targeted marketing data (Shaw et al. 2021), novel survey formats (e.g., interactive surveys; Collins et al. 2012), and targeted sampling frames (e.g., online panels; Circella et al. 2016), to name a few. Another approach, which is the focus of this paper, is to recruit survey respondents who had expressed willingness to be contacted again in a previous survey; this approach has been shown to produce a significantly higher response rate and lower cost per valid response relative to random sampling (Amarov and Rendtel 2013; Kim et al. 2019; Circella et al. 2020).

This recruitment method has some similarities to the approach used in panel studies in that both nominally draw respondents from preceding surveys. Accordingly, both approaches are subject to attrition biases. There are some important differences, however. For one thing, respondents to a panel study are normally informed at the outset that participation in the study involves completing multiple surveys (and therefore that agreement to participate signifies agreement to complete multiple surveys), whereas in the present case, the willingness to complete a later survey is an entirely separate decision, not even presented to the respondent at the entrance to the initial study. Other differences reside in the survey purpose, contents, or outcome. Specifically, panel surveys focus on repeated observations on a set of variables for the same sample unit over time (Lavrakas 2008), which allows the tracking of specific variables or study interests. In contrast, recruiting respondents from a previous survey is not a periodical behavior, and the follow-up survey may have relatively little in common with the initial one. The use of this recruitment method: (1) increases the survey response rates obtained on follow-up surveys; (2) reduces the financial burden for local transportation agencies and researchers; and (3) facilitates the expansion of the variable set of the preceding survey and enables data fusion across datasets (Shaw et al. 2022). In view of the plethora of single cross-section surveys and the challenges of conducting panel studies (notably time and money, among others), using a prior cross-sectional survey to help recruit for the next one is certainly an attractive prospect.

However, in the transportation domain, this recruitment method has not been widely adopted nor carefully examined. A major potential drawback of recruiting respondents based on their willingness expressed in a preceding survey is the non-representativeness that may be inherent in that sample (Couper et al. 2007). Accordingly, the present study is interested in the following questions: (1) Who is more likely to respond to a follow-up survey? (2) How does recruiting respondents based on their willingness expressed in a preceding travel survey bias the follow-up survey sample? (3) In view of the importance (in sample size, geographic scope, and information value) of the National Household Travel Survey (NHTS), how helpful is it to use the NHTS in particular as the springboard for follow-on survey recruitment? Specifically, what survey sample could we expect if we recruited respondents from the 2017 NHTS respondents in different geographic regions in the U.S.?

To address the questions raised above and bridge the gap in the literature regarding recruiting survey respondents from a preceding travel survey, we do the following:

By understanding the dataset biases that can result when respondents are recruited from a preceding survey, researchers/practitioners can better assess the tradeoff between data quality and resource constraints associated with respondent recruitment. Moreover, understanding these biases would allow survey developers to adjust their invited sample—for example, by oversampling underrepresented groups in the follow-up surveys. This work would, therefore, be especially useful for transportation professionals if the NHTS in particular retained the willingness question as a recurring item in future surveys, thereby allowing local agencies and researchers to recruit follow-up respondents from the NHTS sample efficiently. Even outside of the NHTS, the contributions of this paper have general findings and implications for researchers using the approach of recruiting respondents from prior cross-sectional surveys.

The rest of the paper is organized as follows. We begin with the “Literature review” section. We describe the data source used in this study in the “Data description” section. The “Methodology” section introduces details of the probit with sample selection (PSS) model and summarizes six modified model performance measures. In the  “Results” section, we present and analyze model results, including both model interpretations and performance measures. In the “PSS model validation and application” section, we apply the calibrated model to a holdout sample to decompose sample biases and predict follow-up survey participation in diverse geographic regions in the U.S. We close with a summary of findings in the “Conclusion” section.

As mentioned, continuously declining survey response rates make it increasingly difficult for survey developers to obtain high-quality survey data with the same survey budgets as in the past. To enhance response rates, researchers and practitioners have developed and applied many approaches to aid the survey recruitment process.

We first summarize a few commonly used recruitment approaches and the accompanying sample biases. The use of survey incentives is an effective approach to increase survey response rates; examples of these include lotteries, tokens, and philanthropic donations (Edwards et al. 2002; Smith et al. 2020; Young et al. 2020). Coryn et al. (2020) found a lottery to be the most cost-effective incentive format, while Parsons and Manierre (2014) showed that unconditional incentives might exacerbate the overrepresentation of females among survey respondents. Using different survey modes (e.g., mail, phone, and web) is another way to increase response rates of specific population groups. For example, web surveys have (at least in the past) been found to generate a much lower response rate than mail surveys in general (Manfreda et al. 2008; Hardigan et al. 2012), but younger generations such as college students are more responsive to web surveys (Shih and Xitao 2008; Börkan 2010). However, the sample may retain biases associated with the sampling mode, i.e., a mode effect. In a survey aimed at college students, Carini et al. (2003) found that web survey respondents gave more favorable responses regarding computing and information technology than the paper survey respondents. Survey developers could also obtain higher response rates by carefully selecting the sampling frame (Wolf et al. 2005). In recent years, scholars have used commercially-operated online opinion panels, consisting of people who pre-register for survey participation in return for rewards (e.g., cash, vouchers), to reach out to survey respondents and enhance response rates (Neufeld and Mokhtarian 2012; Miller et al. 2020; Chauhan et al. 2021). Some companies that operate these online opinion panels allow quota sampling within the panelists to ensure a (more) representative sample regarding the selected control variables (usually sociodemographic variables). Still, this does not guarantee the representativeness of other variables. For example, a recent study by this team found that online opinion panel respondents have significantly lower life satisfaction than respondents recruited from other sources, even when controlling for socio-demographics (Wang et al. 2022).

Another approach, as previously detailed in the “Introduction”, entails the recruitment of survey respondents who indicated willingness to respond in prior surveys (e.g., Lin et al. 2011). As with the other recruitment approaches discussed, this method also results in unrepresentative samples. Couper et al. (2007) modeled internet users’ willingness to do an online survey and their subsequent follow-up response. They concluded that self-selected samples of internet users are not representative of the population with respect to demographic, financial, and health-related variables. In another example, Germany’s Federal Statistical Office developed an access panel (a pool of persons willing to take part in voluntary surveys) from a large-scale household survey. The access panel was then used as the sampling frame for multiple surveys, and was found to be unrepresentative by multiple teams. Specifically, Amarov and Rendtel (2013) explored the survey participation propensity of the access panel and identified self-selection biases existing in multiple variables, including age, household size, and item-nonresponse. An accompanied simulation experiment (Tobias et al. 2013) on the selection process of the access panel emphasizes the importance of constructing proper statistical models for the access panel recruitment to ensure the appropriate usage of this high-response-rate and low-cost recruitment method. Similarly, Adriaan and Jacco (2009) applied bivariate logistic regressions to analyze the selectivity of the nonresponse of an online panel, which was recruited using a three-stage process: participation in a first telephone interview, willingness to be recontacted, and final agreement to participate in the online panel. The authors found selection biases with regards to age, income, and personal computer ownership.

Although transportation studies on this topic are limited, some studies have examined the nonresponse bias in travel surveys, which could inform the analysis of self-selection biases in recruiting survey respondents from a preceding travel survey. Wittwer and Hubrich (2015) reached out to survey nonrespondents with an abbreviated survey, and found that age and household size have significant differences between main survey respondents and nonrespondents. de Haas et al. (2018) used information obtained from a screening survey and found that age, gender, and education influence people’s willingness to participate in a household travel survey panel. They also found that willingness to participate in a travel survey could modify model coefficients and slightly improve the fits of mode choice models.

This study aims to address the literature gap by examining the practice of recruiting respondents from the NHTS for a statewide travel survey, and constructing a proper statistical model for the recruitment process in the transportation context. We apply the probit with sample selection (PSS) model for analysis, which remedies the selection biases by allowing correlations between the unobservables in the selection and outcome equations (Heckman et al. 2001). The PSS model was proposed by van de Ven and van Praag (1981), which is modified from the Heckman model (Heckman 1976; originally designed for correcting sample selection biases in linear regressions) to fit binary outcome dependent variables. In the transportation domain, sample selection models have been applied for various purposes, one of the most common of which is to correct for residential self-selection effects (Cao 2009; Chen et al. 2017; van Herick & Mokhtarian 2020). In that context, outcomes are observed for both “selected” and “unselected” groups. In other contexts, including ours, outcomes are only observed for “selected” cases—for us, the cases who self-select into both being willing to respond, and actually responding, to a follow-up survey (Alemi et al. 2019; Sun et al. 2019). In this study, we select the PSS model structure since it both fits our data structure (see the “Data description” section) and matches the conceptual reasoning (see the “Model structure and application” section).

The National Household Travel Survey (NHTS) is a repeated cross-sectional travel survey conducted by the Federal Highway Administration, and is widely used by regional planning agencies across the United States. The Georgia subsample of the 2017 NHTS constitutes the survey dataset used for this study. The NHTS typically obtains household, individual, vehicle, and trip information using several survey instruments; these include a recruitment survey, a retrieval survey, travel logs, and a vehicle odometer mileage form. In 2017, for the first time, NHTS allowed states to opt into including a question regarding respondents’ willingness to participate in follow-up travel surveys, and Georgia was one of the six states/regions that chose to do so. We segmented NHTS Georgia respondents based on their willingness to participate in a follow-up survey as well as their actual response behavior to the follow-up survey (see Decisions 1 and 2 in Fig. 1).¹ The follow-up survey, denoted the GDOT survey in Fig. 1, is further discussed later in this section.

As shown in Fig. 1, the first decision was made through the willingness question in the NHTS (i.e., “Would you be willing to participate in a follow-up survey?”). This question is only asked of the main household respondent (i.e., the respondent who answered household-related questions in the retrieval survey), and solely of those living in the regions (i.e., states or Metropolitan Planning Organization areas) that specifically requested the inclusion of this question, with Georgia being one of those regions as mentioned before. As such, we used only the main household respondents for analysis purposes, as we did not have additional information regarding other household members’ willingness to participate in a follow-up survey. The final working dataset comprised 8418 respondents, 4,965 of whom indicated a willingness to participate in a follow-up survey (W1), whereas the remaining 3,453 respondents did not want to be contacted again for future surveys (W0).

For the 4,965 NHTS respondents who indicated a willingness to participate in a follow-up survey, their second decision (Fig. 1) was made through their actual response to a follow-up survey, the Georgia Department of Transportation-funded Emerging Technologies Survey (GDOT survey, Kim et al. 2019). The GDOT survey is a 15-page attitudinally-rich travel survey with an emphasis on the impacts of emerging technologies on travel behavior. Our research team mailed the GDOT survey to the 4,965 NHTS respondents in September 2017. The respondents could either mail the completed paper survey back using the postage-paid reply envelope we provided, or use the URL we also provided to complete the survey online. Ultimately, 1,432 of the 4,965 NHTS respondents replied to the GDOT survey (W1R1), while the remaining 3,533 did not reply (W1R0). Thus, at this point, we have segmented all 8418 NHTS Georgia respondents based on the two decisions. We note that for the purpose of this paper, the GDOT survey was used only to segment/classify respondents; all respondent data was obtained from the NHTS.

In Table 1, we present descriptive statistics for each segment and the overall sample. In the full sample, the average household size is 2.13, the average age is 55.6 years, and 53% of the sample is female. Overall, participants are highly educated, with 48% of the participants reporting they have a bachelor’s degree or higher. Compared to respondents who are unwilling to be contacted (W0), respondents who are willing to be contacted for a follow-up survey (W1) tend to be younger (means of 54.35 versus 57.30 years). On average, the W1 segment conducts more trips on the selected travel day (4.16 versus 3.52 trips) and lives in denser areas (859.07 versus 769.92 housing units per sq. mi.). Among the respondents willing to be contacted, those who replied to the GDOT survey (W1R1) tend to be older than those who did not reply (W1R0, 59.00 versus 52.46 years). The W1R1 segment conducts more trips (4.47 versus 4.03) on the selected travel days, and they come from less dense areas than other groups.

In the following sections, we separate the final working dataset (N = 8,418) into a training set (60%, N = 5,051) and a test set (40%, N = 3,367) to enable appropriate model evaluation.

In this section, we first present the PSS model result (Table 4) and then measure the model performance with the six metrics presented in the previous section (Table 5).

In this section, we will first apply the PSS model to the hold-out NHTS Georgia sample (the test set) to further validate our model results (Parady et al. 2021) and retrieve sample biases in the follow-up survey from multiple sources (Scenario 1, Table 2). We will then apply the PSS model to selected states in diverse geographic regions of the US (California, Massachusetts, Minnesota, North Carolina, and New York) and the full 2017 NHTS national sample, to predict follow-up survey participation and test the transferability of the PSS model (Scenario 3, Table 2).

In this study, we identified and analyzed the self-selection bias existing in follow-up survey respondents who were recruited from a preceding travel survey (the 2017 NHTS). We applied a probit with a sample selection (PSS) model to examine the willingness of NHTS respondents to participate in a follow-up survey, together with their actual response behavior. Overall, as expected, we identified self-selection biases among survey respondents recruited from a preceding household travel survey. Findings suggest that the requirements of the preceding survey influenced respondents’ willingness to participate in follow-up surveys. In the particular context of NHTS, respondents from survey-burdensome households (e.g., large households) were less likely to report being willing to respond to a follow-up survey, although individuals reporting more trips were unexpectedly more likely to be willing. Respondents’ attitudes towards privacy, and some other travel-related characteristics, were also influential to their willingness to be contacted for a follow-up survey. For example, respondents from specific groups (e.g., travel-restricted people, frequent transit users) were more likely to report being willing to participate in a follow-up survey. By participating in travel surveys, these groups may be seeking to improve the quality of their travel. We also found three explanatory variables with opposing signs between the selection and outcome models, a finding that indicated inconsistencies between people’s reported willingness (to participate in a survey) and their actual (response) behaviors. Similarly, the negative error term correlations signified that, on net, unobserved characteristics had impacts on selection that were opposite to their impacts on the outcome.

PSS models do not have model performance measures that are consistently reported in the literature. To address this gap, this paper summarizes six well-known model performance measure categories, adjusted based on the PSS model structure: the log-likelihood, McFadden’s pseudo R-squared, information criteria, point-biserial correlation coefficient, root mean squared error, and success table. McFadden’s pseudo R-squared bounds the model fit between 0 and 1, which is straightforward for understanding and could be used to compare across different PSS models. The success table provides overall model performance measures as well as performance measures for each alternative, which supplies information important to evaluating the model.

We analyzed the representativeness of the follow-up survey respondents regarding 17 selected variables, including sociodemographic and travel-related variables. We decomposed the divergence of the marginal distributions between the population and the predicted follow-up survey respondents into five components, namely dataset bias, household representative bias, self-selection bias, non-response bias, and prediction error. Results showed that the household rep selection contributed to a large proportion of the distribution divergence of individual-level variables. The effect size for marginal distributions showed that education and age were the two least representative individual-level variables in the follow-up survey, whereas homeownership had the largest effect size among the household-level variables.

We also applied the PSS model to different geographic regions of the U.S., namely California, Massachusetts, Minnesota, North Carolina, and New York. Similar effect sizes across states indicated good generalizability of the PSS model, however education, age, and homeownership were still poorly represented among predicted respondents to the follow-up survey for these other states. New York had less representative predicted follow-up survey respondents compared to other states, presumably a consequence of its diverse population composition and different transportation-related lifestyles.

These results can help survey developers assess the representativeness and cost-effectiveness of the proposed sampling frame (i.e., a pool of previous survey respondents), which in turn will suggest adjustments to the sampling frame that can improve the representativeness of the new sample. Specifically, by using this approach to identify likely biases in the follow-up survey sample, study designers may choose to proactively oversample the predicted-to-be-underrepresented groups when recruiting from other data sources (e.g., online opinion panels). We recommend that large-scale travel surveys like the NHTS retain the willingness question as a recurring item, thereby allowing local agencies and researchers to efficiently recruit follow-up respondents from their sample. In fact, we recommend that the question be asked of all survey respondents, not only the main household respondent as was the case here. Recruiting future survey respondents from among all willing preceding survey respondents could substantially reduce sampling biases at the outset.

In a companion study (Wang 2021), we analyze the consequence of self-selection biases by assessing their influence on travel behavior models developed on the second-stage sample. We examine and compare two techniques (sample weights and sample selection models) that could remedy the influence of unrepresentative samples recruited from a preceding survey on travel behavior models.

The study also has several caveats. First, the follow-up survey is a personal travel survey instead of a household travel survey. Our results do not speak to a situation in which the follow-up survey aims to obtain answers from all household members. If “household willingness-to-respond” is defined to be “willingness of every household member to respond”, we would first of all expect a much lower willingness rate, and if follow-through response is required from every household member in order to count, we would secondly expect a much lower follow-through rate among the reported-to-be-willing households. We would further expect more severe biases on the part of the willing and responsive households. For example, our results suggest that, in view of the heavier burden, larger households will probably be less likely to express willingness to respond and to actually respond to follow-up surveys. Given these concerns, we imagine that it would be prudent, if at all possible, to allow something less than full household participation to “count”, at both stages of the process. Nevertheless, it is not presently clear how best to balance the disadvantages of a smaller and more biased sample when requiring full participation, against the disadvantages of incomplete household information when relaxing that requirement.

Another caveat is that the follow-up survey lags the preceding one by an interval ranging from four to 18 months, during which the address and demographic information of the initial survey respondents may have changed without our knowledge. We encourage future studies to explore the impact of time interval on the actual response to follow-up surveys. Moreover, it can be interesting to study the impact of completion modes (e.g., paper, online) for both preceding and follow-up surveys on the willingness to participate.