Alternative selection mechanisms in online surveys

Online samples can take various forms, but they all share the common feature that participants are contacted, or responses are collected, via the internet at some point, see Callegaro et al. (2015) and Vehovar et al. (2016). Persons may first be selected through a probability sample and are then asked to complete an online questionnaire. See Amarov and Rendtel (2013), Enderle et al. (2013), and Rendtel and Amarov (2014) for applications within the framework of the German Microcensus, or consider the German Internet Panel of the University of Mannheim (Blom et al. 2015). However, there are approaches where internet users are asked by some widgets to participate in some voting experiment, e.g. how they judge an actual event in an online newspaper. Those who participate in the voting experiment are then asked for further participation in a survey. This kind of selection is called “River Sampling”(AAPOR 2010).

Despite uncontrolled self-selection effects (Bethlehem 2010), online surveys via River Sampling have become popular survey instruments as they deliver cheap and fast results. Moreover Elliot and Valliant (2017) have produced the framework of Quasi-Randomization (QR), which under regularity conditions, allows an unbiased estimation of population parameters from non-probability surveys. For a given set of variables, this approach compares the distribution of a non-probability sample with the distribution of a probability sample. If these variables explain the self-selection mechanism of the non-probability sample, the weighted results from the QR-approach deliver valid population estimates. The draw-back of this approach is the unobserved selection process, as non-respondents are not recorded in the River Sampling procedure. In practice, one uses demographic variables, like age groups, gender, education and regional indicators which are known for the non-probability and the probability survey. This makes the computation of QR-weights feasible.

In order to evaluate the approach, one has to use a simulation strategy where the self-selection from the population is repeatedly simulated. Hence, we need a survey which contains information on internet use, on demographic variables and on the outcome variables which are of interest. By using variables on internet use we aim to generate a plausible selection process, which yields an artificial self-selected sample. Finally, we compare the results of the outcome variable from the original sample and the self-selected sample. Differences indicate how well the QR-weights reduce possible self-selection effects.

Schnell et al. (2017) were the first authors who used the European Social Survey (ESS) for such an evaluation. Their main outcome variable is a reported health status. For the self-selection process they generated a dummy variable, which assumes self-selection if the person has access to the internet and uses it. They used ESS data from 2010. They conclude:

“Weighting by calibration on age, gender, ethnic background, urban residence, education and household income does not eliminate the observed health differences. Therefore, the underlying missing data mechanism might be considered as an example of MNAR¹ If this holds, no weighting strategy will be able to eliminate health bias in web surveys”. (Schnell et al. 2017).

However, internet access and its use has dramatically changed since 2010. Now, it is present in almost every household in Germany (Qualitätsbericht EVS 2018; Statistisches Bundesamt 2022)² and its use is no longer restricted to PC’s. Easy-to-use apps on smartphones can be utilized almost everywhere in public. Additionally, distinction according the general definition of the so-called “digital divide”: “a division between people who have access and use of digital media and those who do not”(Van Dijk 2019), is undergoing change. Therefore, we expect that results related to selection effects in online surveys from 2010 cannot be transferred to more recent years.

This article aims to formulate more adequate models on current internet use. This includes individual usage styles, like posting on social media, and motivational factors (Bethlehem 2010). As a framework we use the ESS data from 2018³. For bias correction we use the QR-approach and calibration to population totals which is common among survey practitioners. We use the votes for political parties in the previous general election of the German Bundestag 2017 as outcome variable and analyze how well the bias-corrected party percentages meet the simulated values of the ESS-universe. In order to scale these differences, we compare them to the differences of original ESS-estimates and the official election results. We will also discuss selection models where we include variables which are closer to the outcome variable than the standard demographic variables.

In the early days of the internet survey, statistician were concerned about the proportion of the households with access to the internet. For example, in Germany this percentage was as low as 8.1% in 1998 (Statistisches Bundesamt 2022). An example provided by Valliant (2020) pertains to the Michigan Behavioral Risk Factor Surveillance Survey (BRFSS) conducted in 2003, which served as a basis for simulating selection via internet access. According to Valliant, the internet coverage rate for this survey was 60%. However, the internet coverage has dramatically risen in the last two decades. Within 10 years the internet coverage in Germany has risen from 8.1% to 64.4% (2008) and in 2018 to 92.7% of the German households.

While internet access has become apparently less important for participation, the duration of daily internet use can be a good indicator to measure the probability of coming across invitations to participate in an internet survey. The longer the duration, the higher the probability of coming across an interesting widget.

Early on, the internet was merely a tool for emails and reading pre-made HTML documents. However, over the years, the purposes of internet usage have become more diverse, see van Eimeren and Gerhard (2000), with activities such as “mainly reads online articles”, “tends to write emails”, or “plays online games”. Hargittai (2002) referred to disparities based on online skills as the “second-level digital divide”, which influences how people use the internet.

Nowadays, aspects of interactivity seem to be important for online survey participation. Multiple studies (Chang and Krosnick 2009; Dever et al. 2008; Malhotra and Krosnick 2007) have reported that active internet users tend to be over-represented in online panels. A good indicator of active use is the online posting behavior.

Motivational aspects may also be important for the self-selection (Bethlehem 2010). Chang and Krosnick (2009) found that political participation in the United States is approximately 10% higher in online samples compared to telephone or probability-based internet samples. Therefore, if the outcome variable is voting behaviour, a recent posting about politics could be a relevant motivation to follow an online survey about political party preference.

A different aspect of online surveys are incentives to participate. A traditional incentive is a payment. Often such payments are linked with the recruitment to an access panel, which cumulates the information of its respondents from the separate surveys (Amarov and Rendtel 2013). A different reward which is specific for online surveys is the immediate return of the voting of the other participants on the topic of interest. For example, the German online survey agency Civey poses its widgets on daily emerging topics of relevance in online newspaper platforms, readers interesting in these topics are probably keen on learning about the views of others on the matter, see Richter et al. (2023) for details. Thus, it makes sense to use an indicator of interest, for example, interest in politics, in a participation model for online surveys. However, one should not ignore the risk of topical self-selection (Lehdonvirta et al. 2020), which is inherent to this approach. This suggest to use interest in the topic and curiosity about the opinion of others participants as a means to contact potential respondents. But one should refrain from using these responses as valid answers which are representative for the population. Therefore Civey does not use these first contact responses which are framed by an actual event in a newspaper, see Richter et al. (2023). Nevertheless, interest in the topic seems to be a reasonable indicator for participation in online surveys.

The use of survey weights is the standard procedure for the design-based estimation of population totals and proportions, see, for example, the textbook of Särndal et al. (1992) or Lohr (2023). In this section we shortly discuss two weighting approaches which can be also used for non-probability surveys. Elliot and Valliant (2017) designed the quasi-randomisation (QR)-approach especially for the estimation of population parameters from non-probability samples. Calibration has a long history in survey sampling, with fundamental contributions from Deming and Stephan (1940), Deville and Särndal (1992), Deville et al. (1993). It serves as a standard tool to correct nonresponse bias. A second important issue is the reduction of the variance of population estimates, see (Lundström and Särndal 2001) or the overview of Särndal (2007). In the context of non-probability samples, Yang et al. (2018) demonstrate an application where units come from an opt-in online sample.

Both procedures implicitly model selection probabilities such that the resulting survey weights apply to all outcome variables. This is an advantage over model-based approaches which predict the value of the outcome variable for the non-observed units, see Little and Rubin (2020) for a model-based treatment of missing values.

The basic approach is to use a high quality survey sample for the generation of an artificial population. We choose the German subsample of the European Social Survey (ESS) as a database for our simulation experiments. In this population we have individual information on the internet use, interesting outcome variables and variables relevant for the above weighting procedures. We formulate a plausible model of the selection process. Then we will select repeated samples from the population. On the basis of each sample we apply the above mentioned weighting procedures. In the evaluation step we compare the population values with the uncorrected sample results and the weighted sample results. Thus we see a potential bias and also its potential reduction by weighting. As we also know the value of the outcome variable we, can also check the MAR condition⁴. Thus we can check whether standard demographic variables are sufficient to explain the participation process which are formulated in variables of internet use. Variances and confidence intervals can be derived via replications of the sampling process. An early application of this design was used by Schnell et al. (2017).

A substantial advantage of this approach is that effects of a mode change are avoided. Thus, the displayed biases are solely due to the selective effect of internet use. Standard comparisons of online surveys and surveys with different interview modes, like in Cornesse et al. (2020), are always confounded with potential mode effects, see Bottoni and Fitzgerald (2021) for results from the ESS.

We first display the development of access to the internet according to the ESS data. We will then present a trend analysis on the bias of two outcome variables which is due to a simple restriction to internet users, i.e. all ESS persons who reported an access to the internet remain in the analysis while all persons with no access or no reported use of the internet are discarded from the analysis. The first outcome variable is the subjective health index which was used by Schnell et al. (2017). The second outcome variable is the reported vote in the election of the German parliament. In the main part of our analyses, we first check how well the demographic variables fit within the framework of the selection mechanisms SM‑1 to SM‑4. Then we investigate the reduction of bias by the QR-weights and the calibration procedure. Finally we scale the bias relative to the difference of the ESS estimates and the real election result for the 2017 Bundestag.

We have demonstrated the use of the ESS as a simulation basis for the evaluation of weighting strategies for data from online surveys. The selection process of the data is assumed to be not observed, like in River Sampling from the internet. Therefore we formulated four potential selection models which are motivated by a‑priori knowledge on internet use: the duration of daily internet use, the activity in social channels via posting, and the interest in politics.

We used two weighting strategies which do not rely on design-based sampling probabilities. First the Quasi Randomization (QR) approach which compares the non-probability sample distribution with a probability reference distribution, and second, the calibration approach, which restricts the weighted sample results to known population values. These weighting procedures use variables which have to be recorded in the non-probability sample and also in the reference sample or population. For this reason, the set of variables is often restricted to demographic variables. However, if available, additional variables which are potentially linked to the selection process, can be used too. Therefore we used life satisfaction, trust in people, and political partisanship. The latter variable seems to be especially useful for the prediction of survey participation in an online survey on politics. As an outcome variable we used the votes for the previous parliament election in Germany.

In this simulation setting, we could test the MAR-condition. In only one out of four for selection models the demographic variables were sufficient to guarantee the MAR-condition. This couldn’t be improved by the use of additional non-demographic variables, like political partisanship. However, despite whether the MAR-condition holds or not, we achieved substantive bias reductions in both cases.

We compared the ESS population estimates for the 2017 Bundestag election with the real election result. We used the difference of this high quality probability survey to the population count as a means to judge the differences in our simulation setting. It turned out that the total absolute differences of the original ESS estimates and the population results were about the same size as the uncorrected biases of our four selection schemes. The weighting procedures did reduce these differences in some cases by more than a half of their initial size. Thus, we obtained a better fit of the population values than the ESS in the case of the Bundestag results.

We have also shown that previous selection models which are based on the mere access to the internet, either defined by no internet connection at home or by non-use, have meanwhile lost their potential to create a substantial bias. For this result we have used the biannual replication of the ESS from 2002 to 2018. Therefore, previous results on internet online survey which are based on mere access are probably no longer valid.

We choose our selection models on general plausibility arguments of participation in surveys. In these models a selection bias can be controlled to a size which is not bigger (or even smaller) than in a high quality survey. The weakness of our approach is the lacking empirical knowledge on the selection process. Therefore, it is important to further investigate potential determinants associated for online participation. Survey agencies conducting online surveys might run experiments with selected subgroups of their access panel participants and their information on internet use.

A different useful dimension of the ESS which has not been used here is the comparability over its European subsamples. For the analysis presented here we have only used the German subsample. However, it would be interesting to investigate our sampling models also for the other European participants of the ESS. In this case, one should use a variable with an overall general meaning, like the satisfaction with health.