Asynchronous fieldwork in cross-country surveys: an application to physical activity

We use data from ESS Round 7 European Social Survey Round 7 Data (2014). ESS is a cross-national survey exploring various socio-economic circumstances and attitudes of Europeans in over 20 countries. The ESS sample is a probability sample drawn for each participating country, covering different sampling methods, ranging from stratified random samples to multi-stage random samples, aiming to collect national samples representative of their population aged 15 and over. Interviews are normally conducted using face-to-face computer assisted personal interviewing, in addition to a self-administered questionnaire and showcards.

Despite challenges arising in conducting cross-national surveys, ESS manages to implement a harmonized cross-national survey which maintains a high-quality survey design, fieldwork monitoring, and survey management since the first round (administered in 2002) and all subsequent rounds every two years. Compliance profiles with metadata and paradata are regularly published to inform researchers about the fieldwork periods, interviewers composition, contact, and response rates.

As often the case in cross-national surveys, different professional survey agencies are responsible to undertake the fieldwork in each country, under a general set of common requirements across countries in order to ensure the quality standards of the fieldwork procedure and survey design for specific fieldwork periods within each round. ESS Round 7 responses were between 2014 and 2015.

We choose ESS Round 7 as it is the most recent round which contains a measure for physical activity. Physical activity was measured in terms of how many days did the participant played sports, walked quickly or did any other physical activity for 30 min or longer, in the last 7 days. The date (day, month, and year) of the interview was recorded by the interviewer. We use the date in our process of selecting which countries will be included in our analytical sample. In our analysis, we use months only. Information was also collected on respondents’ sex, age, their education measured by the International Standard Classification of Education (ISCED) level, the income decile they belong to, and whether they live with a partner or children. To demonstrate that asynchronous fieldwork may lead to biased differences of physical activity across countries, we select six countries such that their national samples were all collected within the exact same period, and no country had a month with zero observations. Other countries extended the collection period beyond January 2015, while other countries had zero collection for some months within the September 2014–January 2015 period (i.e. starting later or finishing earlier). This resulted in selecting Belgium, Switzerland, Germany, Finland, Ireland, and the Netherlands.

We use linear regression analysis to demonstrate the implications of not accounting for asynchronous fieldwork between countries. We use interview month dummies to capture the seasonality. We fit unadjusted models using country dummies only, and using country dummies with month dummies. Adjusted estimates were used controlling for age, sex, income decile, ISCED level, living with a partner, and living with children. We employ unweighted regression and weighted regression using design weights provided by ESS to account for differences in sampling design.

In post-estimation, we make country-specific predictions using the models with month dummies. We provide three different types of predictions. The first one is a baseline and uses the sample means of month dummies in each country (which can be obtained in Stata using margins, over(country) atmeans). These predictions are similar to the ones from the model without month dummies. The second uses the pooled means of month dummies for all countries, i.e. assuming the same distribution of months for all countries (which can be obtained using margins, over(country) atmeans at(1.month=.25 2.month =.29 3.month=.21 4.month=.13 5.month=.12), with the values taken from the pooled means of the month dummies in our analytical sample). The third uses a balanced allocation, i.e. assuming an equal number of observations in each month (which can be obtained using margins, over(country) atmeans at((asbalanced) month) or equivalently in our example: margins, over(country) atmeans at(1.month=.20 2.month =.20 3.month=.20 4.month=.20 5.month=.20)). The choice between the last two types depends on the practitioner’s preference. In either case, the difference between any two countries remains the same, since the model is linear.

Table 1 reports the characteristics of our analytical sample. Our sample is consisted of 12,663 respondents across the six countries, with Germany having the largest sample, and Switzerland the smallest. Most participants were interviewed during the Autumn months (September, October, and November), while about a quarter were interviewed during the Winter months (December and January).⁴ Overall, respondents reported a mean of 3.5 days of physical activity in the last 7 days before the interview, with a standard deviation of 2.5 days. Table 1 also reports descriptives of the variables used in the adjusted regression models, namely age, sex, income decile, ISCED level, living with a partner, and living with children.

We illustrate the asynchronous fieldwork across countries and interview dates graphically in Fig. 1. The left-hand side graph shows the cumulative frequency of interview date for each country, which represents the rate of the data collection during the fieldwork period. A line that is steep at the beginning and then flat suggests that the country conducted many interviews at the start of the fieldwork period, whereas an initially flat and progressively steep line suggests that the country delayed the collection of most interviews at the start of the fieldwork period. Thus, we observe that Switzerland advanced much faster the data collection, followed by Belgium, Finland, and the Netherlands, while Germany and Ireland lagged behind. Figure 1 also shows on the right-hand side the distribution of interview months across countries in percentages. It is evident that, although within a common fieldwork period, countries performed the fieldwork in very different patterns. For example, Switzerland started the data-collection process fast and within two months covered almost 80% of its target sample, whereas Germany and Ireland had only about 40% in that same period.

Bild vergrößern

Table 2 shows the results of the unweighted regressions and demonstrates how the country coefficients change after the inclusion of month dummies. Columns (1) and (2) report the estimations without controls, while columns (3) and (4) present results from estimations using the controls described in the previous section.

The coefficients of the month dummies reveal that September is the month with the highest number of days doing physical activity. The gradually increasing magnitude as month goes towards January indicate that physical activity is being less and less prevalent in the winter months. Moreover, the coefficients of the country dummies change, especially for Switzerland. A joint test on the coefficients of the month dummies also provides a statistical test of the importance of asynchronous fieldwork in the outcome of interest. In this model, an F test gives a value of \(F=4.07\) with p value 0.003, rejecting being jointly zero. The magnitudes of the month dummies are also indicative of the scale of difference one can expect in the predictions when accounting for interview month.

Table 3 presents the three types of post-estimation predictions: the baseline prediction, the pooled prediction assuming the same distribution across months, and the balanced prediction assuming balanced distribution across months. Using the model without controls (Columns (1)–(3)), we can compare, for example, Switzerland and Germany. The baseline prediction results in similar levels of physical activity for these two countries (3.726 for Switzerland and 3.716 for Germany); however, if we account for interview months we see a statistically significant difference of 0.07 (\(p<0.05\)) in the model with controls, with Germany now surpassing Switzerland (column (5) in Table 3).⁵ This is due to the prediction for Switzerland going down, as it had a larger share of early months (September/October), and the prediction for Germany going up, as it had a larger share of later months (December/January). These changes are also observed in the weighted regressions, which show similar trends in the month dummies and similar directions in the change of predictions (Tables 4 and 5 in the Appendix, using the provided calibrated weights and design weights, respectively).