ICT skills measurement in social surveys: Can we trust self-reports?

Skills are an important aspect of human capital, which is, in turn, one of the main factors necessary for economic and social development. In addition to economic growth discussions, human capital and skills measures are often embedded in discussions on the economics of education, ageing, employability, crime, well-being, and health (e.g., Stroombergen et al. 2002). Skills may be analysed from different perspectives: namely, supply and demand, skills mismatch, and skills development (European Commission and Statistical Office of the European Union 2016). Economists have investigated the returns to both cognitive and non-cognitive skills with respect to labour market outcomes and social behaviour (e.g., Heckman et al. 2006).

Estimates of individual or population skill levels are typically obtained using direct or indirect measures. Indirect measures of skills supply include information on the level, orientation (general or vocational), and field of educational attainment; i.e., the information that can be obtained from administrative registers or from social surveys, including the Labour Force Survey. Direct measures include self-assessments of skills (e.g., Adult Education Survey), self-reports of the ability to perform specific tasks (e.g., ICT Usage in Households and by Individuals, Skills and Jobs Survey), and objective skills assessment frameworks (e.g., Programme for International Student Assessment—PISA or Programme for the International Assessment of Adult Competencies—PIAAC) (European Commission and Statistical Office of the European Union 2016).

Here, we concentrate on skills related to ICT, and specifically on the ability to perform basic tasks on a computer. These skills have become increasingly important in people’s daily and professional lives since the computer and computer network revolutions began in the 1970s and 1980s. Thus, the early development of ICT or computer skills is increasingly perceived as being as critical as reading and writing; and has even been called a “new literacy” (Grant et al. 2009). ICT knowledge and usage are among the digital skills frequently cited as being necessary in the twenty-first century (e.g., van Laar et al. 2017). Levels of ICT skills and of computer and internet usage across populations are of interest to international organisations and institutions, as well as to national governments, as the numerous strategic documents published on these topics demonstrate.

Scientific analyses have attempted to quantify the gains associated with the acquisition of ICT skills. The results on individual returns to ICT skills in the labour market are mixed, with some studies showing substantial gains (Falck et al. 2016; Krueger 1993), and others questioning the causal link between ICT skills and higher earnings (Borghans and ter Weel 2004; DiNardo and Pischke 1997). As Falck et al. (2016) have pointed out, most of these analyses relied on self-reported measures of computer use and computer skills, which are rather imperfect proxies for the true skills level due to reporting biases, or to individuals’ lack of awareness of the skills distribution at the population level, and of where they are positioned within it.

Studies that compare self-assessments of ICT skills with analogical objective direct assessments have been limited. While some of these studies have shown that the problem of skills overestimation or overreporting is common, their findings referred only to groups of students at selected US or UK universities (Grant et al. 2009; Larres et al. 2003; Merritt et al. 2005). Our study has a much broader scope: namely, to analyse the phenomenon of the misreporting of ICT skills across the whole working-age population of Poland. To the best of the authors’ knowledge, there are no existing studies that examine the quality of ICT skills indicators and the problem of misreporting using a statistically representative sample of a population. Moreover, we use measures for self-assessments of ICT skills administered in international Eurostat surveys. Our results show that the problem of misreporting is prevalent across the surveyed group, but that different subgroups have different probabilities of misreporting their skills. Specifically, we find that people who belong to groups with higher average skill levels are also more likely to overreport their skills when they fail to have them. This finding indicates that the observed discrepancies could be driven by the social desirability bias: individuals who belong to groups with higher skills probably feel that they are expected to have higher skills, which could make them more likely to overreport their skills. The results of our analyses can help ensure that ICT skill levels are properly assessed at the population level, and that governments are able to develop public policies that promote social and economic growth.

The article is organised as follows. In the following section, we present a review of the literature on why respondents misreport. In the data and methods section, we introduce the Polish follow-up study to PIAAC (postPIAAC survey), in which ICT skills self-assessments were followed up with direct assessments of comparable skills. In Sect. 4, we outline the results of the analysis that models the probability of skills misreporting. In the final section, we discuss our results and conclusions.

In surveys, there are many sources of non-sampling errors, including response and measurement errors. Respondents may provide untrue or incorrect information, which may result in biased estimates, for which also the variability may become impossible to assess. The process of answering questions can be seen as a multi-stage process. The Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System (Eurostat 2004) has identified a five-stage process of answering questions: encoding, comprehension, retrieval, judgement, and reporting. Each stage is subject to certain cognitive biases or distortions that may cause the response to deviate from the true answer. Some of these biases can be reduced by following appropriate statistical guidelines related to the survey mode, interviewer behaviour, question formulation, response formats, and approaches to dealing with sensitive topics or fear of disclosure. Three cognitive biases in particular are potentially important for reporting computer skills: information deficits, social desirability, and social approval. The first of these biases relates to the stages of comprehension, retrieval, and judgement; while the other two biases relate to the reporting stage.

Dunnig et al. (2004) have provided a comprehensive overview of the psychological mechanisms that lead people to hold opinions of themselves that diverge from objective reality. People typically do not possess the full information that would allow them to engage in perfectly accurate self-assessment. Moreover, individuals often neglect or give too little weight to valuable information that could help them shape a more realistic view of themselves. Inaccurate self-assessments often occur because “the skills needed to produce correct responses are virtually identical to those needed to evaluate the accuracy of one’s responses” (Dunning et al. 2003, p. 85). This effect drives the so-called double curse of incompetence that has been observed in a number of significant social and intellectual domains (Kruger and Dunning 1999); and that can shape individuals’ careers and life course through their impressions of their own skills, talents, knowledge, or personality (Ehrlinger et al. 2008). The information deficits explanation is consistent with the results of a meta-analysis presented by Falchikov and Boud (1989), which showed that students in advanced courses provided more accurate self-assessments than students in introductory courses. Gibbs et al. (2017) went beyond the academic environment to analyse information deficits in the context of workplace end-user computing. Their study was among the first to investigate the existence of the curse of incompetence in the computing domain. Their results supported the existence of this bias.

The other cognitive biases mentioned above include social desirability and social approval. The first of these biases refers to a tendency to avoid criticism and to portray oneself in a favourable social light, whereas the second is defined as the need to obtain a positive response in a testing situation. Social desirability is related to the concept of “injunctive social norms” (Cialdini 2007), which has been defined as an individual’s perception or expectation of what most others approve or disapprove of. Social desirability effects are likely to exist when sensitive questions are asked, and are more likely to be observed in interviewer-administered than in self-administered surveys (Aquilino 1994; Aquilino and Lo Sciuto 1990; Felderer et al. 2019; Gnambs and Kaspar 2017; Kreuter et al. 2008; Tourangeau 2018). In addition, time use analyses have shown that in many cases, the diary method of data collection is more accurate than the use of questionnaires (Bonke 2005; Kan 2008; Niemi 1993). However, in a literature review, Bosnjak (2018) found no strong evidence of differences in misreporting tendencies across modes. Yet it should be noted that the understanding of what constitutes a sensitive question can differ across individuals and modes of data collection (Kreuter et al. 2008).

The social desirability and social approval mechanisms have, for example, been investigated in comparisons of self-assessed and objective measures of physical activity. In the U.S. context, estimates of the proportions of Americans who adhere to the official guidelines regarding the time people should spend on moderate to vigorous physical activity vary significantly across studies that rely on different approaches to collecting information. For example, using both objective and subjective information on physical activity, Adams et al. (2005) found a positive association between the social desirability of an activity (as measured on a specific scale) and the tendency of people to overreport the amount of time and energy they spent on the activity. Unexpectedly, they also detected a correlation between social approval and the tendency to underreport physical activity, but the magnitude of this association was small. Several other studies have stressed the role of social desirability in the overreporting of physical activity, and have found different propensities to overreport for subgroups distinguished by sex, age, education, weight, and other characteristics (Beyler and Beyler 2017; Ferrari et al. 2007; Hill et al. 1997; Nusser et al. 2012; Sallis and Saelens 2000; Troiano et al. 2008); or by the data collection procedures used (Brenner and DeLamater 2016, 2014).

It has also been observed that energy intake or diet survey data are subject to measurement errors. The self-reported data on food intake are usually biased downwards. This bias has been related to social desirability scores, and appears to differ by sex, eating habits (amount of fat consumed), and, potentially, education (Hebert et al. 1995, 1997, 2002). In addition, there is evidence that socially desirable answers are often given when reporting voting behaviour (Bernstein et al. 2001; Holbrook and Krosnick 2010) and religious service attendance (Brenner 2017; Brenner and DeLamater 2016). In these cases, overreporting is usually observed. Underreporting tends to occur when people are asked about contra-normative or embarrassing behaviours, like drug use or nonregistered employment (Aquilino 1994; Aquilino and Lo Sciuto 1990).

A vast body of research has examined the quality of self-reported measures on health. Johnston et al. (2009) investigated the reporting heterogeneity on hypertension in England. They found that the probability of false negative reporting was significantly income-graded, which implies that the true income-related inequalities in health were underestimated. Among the reasons cited for health status misreporting are that people have different concepts of the meaning health; that individuals’ expectations of their own health and their use of health care can vary; and that people may have difficulties comprehending the survey questions asked. The last issue may be corrected by the use of the anchoring vignettes methodology, and was applied by Bago d’Uva et al. (2008) in correcting the disparities in health by education and income observed in selected Asian countries.

Finally, some studies have shown that personality and individual differences are related to the tendency to misjudge one’s own skills. MacIntyre et al. (1997) investigated the relationship between individuals’ perceived competence in a second foreign language and their actual competence, as well as their language anxiety. They found that a high anxiety level was correlated with a tendency to underestimate one’s skills, while a low anxiety level was correlated with a tendency to overestimate one’s skills. Anthony et al. (2000) and Korukonda (2005) analysed the relationship between computer anxiety, technophobia, and personality. They showed that neuroticism was positively correlated with technophobia, while openness was negatively correlated with it, which might affect the self-assessment of ICT skills. Moreover, Judge et al. (2002) showed that neuroticism, locus of control, self-esteem, and generalised self-efficacy were strongly related; and that a single factor explained this relationship: namely, core self-evaluation. It might be expected that individuals who have lower core self-evaluation levels (and are thus more neurotic) tend to underestimate their own skills.

Only a few studies have looked at the discrepancy between self-assessed and objective measures of computer skills. Grant et al. (2009) compared self-assessments through questionnaires with computer-based assessments of ICT skills among more than 200 business students enrolled in an introductory business computer applications course at a US university. Their results showed that the students tended to overestimate their ICT skills, and especially their spreadsheet and word processing skills. No differences were found between the students’ perceptions of their presentation skills (MS PowerPoint) and their actual performance. Larres et al. (2003) reported similar results for accounting students at two UK universities: i.e., they found that a majority of the students overestimated their computer literacy. Analogical findings have been reported for students in the US by Merritt et al. (2005). However, none of these results can be generalised to a total adult population, and none of these studies investigated the differences between the groups or their potential reasons for overreporting ICT skills.

The share of individuals who were able to perform the tasks differed significantly across the tasks. The spreadsheet task was more difficult than the file managing and the text editing tasks. The spreadsheet task was completed by 24% of respondents who had ever used a computer, while the file editing and the text editing tasks were completed by 73% and 74% of respondents, respectively (Table 1). The overreporting of skills was much more common than the underreporting of skills. For all of the tasks, most of the respondents who had the respective skill correctly reported it, while among the respondents without a given skill, half or more overreported their skill level. The classification measures are included in Table 2. Respondents correctly reported their skills in the most difficult task (spreadsheet calculations) less often than in the two easier ones (53% versus 79% and 81%). Only 32% of those who reported using spreadsheet calculations before actually completed this task while this proportion was 82% and 84% in case of the other tasks.

The proportions of individuals who could perform simple calculations in a spreadsheet, and who overreported their skill levels if they could not, differed by their socio-demographic characteristics (Table 3). The results for file managing and text editing tasks reveal different levels of skills, but follow a similar pattern of differences between the groups, and are presented in Tables 8 and 9 in “Appendix”. Whereas 35% of the youngest group (aged 19–24) completed the task, < 5% of the oldest age group (aged 65 +) did so (F(1, 79) = 59.58, p = 0.000).¹ The propensity to overreport one’s skills decreased with age, from 85% in the youngest group to 35% in the oldest group (F(1, 79) = 31.60, p = 0.000). More men than women completed the task (28% versus 20%) (F(1, 79) = 17.73, p = 0.000), but there was no difference in their propensity to overreport (F(1, 79) = 0.06, p = 0.801). The fraction of individuals with this skill increased with the educational level, while the propensity to overreport decreased. The exception was the group with the lowest level of education (ISCED 2), because many of these individuals were still in education. The share of individuals who completed the task differed depending on whether they were living in a village or in a city. The size of the city did not affect the likelihood of completing the task, but the propensity to overreport increased slightly with the size of the city from 58 to 76% (F(1, 79) = 6.48, p = 0.013). The individuals who participated in a nonformal or informal educational activity had twice as high likelihood of completing the task (F(1, 79) = 66.72, p = 0.000 and F(1, 79) = 59.34, p = 0.000, respectively) and a higher propensity to overreport their skills (F(1, 79) = 80.61, p = 0.000 and F(1, 79) = 145.81, p = 0.000, respectively). Unemployed and inactive individuals were less likely than employed individuals to have completed the task (18% versus 24%) (F(1, 79) = 5.77, p = 0.019 and F(1, 79) = 8.81, p = 0.004, respectively), but the unemployed individuals were more similar to their employed counterparts when it came to their overreporting propensity: over 60% in both groups (F(1, 79) = 0.18, p = 0.675).

In interpreting these results, it is important to keep in mind that the presented estimates characterise the population who ever used computer. If we included individuals who had never used a computer as being without a skill, the differences in the groups’ skill levels would increase.

The main result of the multivariate analysis of the probability of overreporting one’s skills is that those individuals who belonged to the groups with the highest average ICT skills levels were also the most likely to overreport their skills. The propensity to overreport decreased with age and increased with years of education and numeracy level. The analysis also showed that women were less likely than men to overreport their skills (Table 4). These observed gender differences are consistent with the results of Hargittai and Shafer (2006), who analysed the internet-use abilities of randomly selected US internet users, and showed that while the interviewed men and women did not differ much in their online abilities, the women’s self-assessed skills were significantly lower. In general, the level of overreporting was higher in groups who had a higher probability of completing the task: i.e., among men and among the younger and better educated (Table 5). Those characteristics have been shown to be positively related to the observed ICT or internet usage skills in other studies as well (van Deursen et al. 2011; van Deursen and van Dijk 2009). The positive relationship found between the probability of a group to overreport their skills and the expected levels of skills suggests that social desirability bias may explain the overreporting phenomenon. The positive relationships observed between the propensity to overreport and numeracy and years of education call into question the information deficits explanation for overreporting suggested by the literature.

Some of the personality traits were also found to be related to the propensity to overreport. Neurotic individuals were less likely to overreport their skills; probably because they are less self-confident. Other studies have also shown that neuroticism is positively correlated with technophobia (Anthony et al. 2000; Korukonda 2005), which could affect self-assessment of ICT skills. By contrast, perseverance of effort was found to be positively related to the propensity to overreport. A potential explanation for this result is that individuals with high levels of perseverance of effort tend to have high expectations of themselves.

The relationships of overreporting with gender, numeracy, and personality traits were found to be significant only for the spreadsheet task, probably because this task had the biggest sample size (i.e., the number of respondents who did not complete this task). For the other tasks, the estimates had the same direction and the same order of magnitude (see also Table 10 in “Appendix” with linear probability model estimates).

Our results showed that individuals who belonged to groups with higher ICT skills were also the most likely to overreport having these skills. The propensity to overreport one’s ICT skills decreased with age and increased with years of education and the size of city where the individual was living. These results might suggest that social desirability plays a big role in the process of forming the answers to questions related to ICT skills: i.e., individuals who belong to groups with higher skills probably feel that they are expected to have higher skills, which could make them more likely to overreport their skills. This assumption is consistent with the observation of Bernstein et al. (2001) related to the overreporting of voting behaviour: “people who are under the most pressure to vote are the ones most likely to misrepresent their behavior when they fail to do so” (p. 24). As our findings indicated that the propensity to overreport was positively related to numeracy level and years of education, it appears that information deficits did not play a significant role in skills reporting, or their effects were outweighed by the social desirability mechanism.

This study is not without limitations. First, the interviewer effect is usually smaller in the self-administered mode than in the interviewer-administered interview. Therefore, it is likely that the observed discrepancies in the ICT skills measures (subjective vs objective assessment) were reinforced to some extent by the mode effects (interviewer-mediated in the self-assessment in our study versus self-administered in the direct measurement in our study). However, as we were not able to disentangle these two effects, the presented results capture both effects. Future experiments could compare self-administered self-reports with direct assessments, which should eliminate the pure mode effect. Second, a partial explanation for the failure in the direct assessments could be that the respondents used the interviewer’s laptop instead of their own computer with familiar settings. This could have made completing the tasks more difficult for some of the respondents but this effect should have been similar in different socio-demographic groups. Finally, this study differs from typical cross-sectional studies with skills self-assessment in one important respect. The postPIAAC respondents were cognitively tested in the PIAAC survey three years earlier, so they could potentially suspect that this survey also includes a competence test. If it affected their answers to self-reported questions on skills, it would probably limit the incidence of overreporting of skills and our estimates of this phenomenon could be considered as the lower bound of measurement in a typical survey.

Generally, our results support the development of methodologies in which skills are measured directly. The examples of such projects are the OECD’s PIAAC survey. Direct measurements of skills may be more comparable across countries and for some types of skills (like basic digital literacy). Moreover, the use of this approach may be the only way to compare the skills of different populations. Figure 1 shows the discrepancies between the PIAAC’s and the Eurostat’s ICT indicators of computer skills on the country level.

This comparison is limited to the age group 25–34² (i.e., young individuals who are supposed to have the highest skill levels) and to countries for which the data were available in both surveys. In some countries, a significant share of PIAAC respondents refused to take the computer-based assessment (CBA), even though they had reported earlier that they had computer experience. The country with the highest share of such respondents was Poland. Nearly 30% of the PIAAC respondents aged 25–34 did not take the CBA in Poland, while in the Eurostat ICT survey, only 15% of these younger respondents reported not having computer experience or not having done any of the six basic ICT activities. This pattern differs by country: for example, in the UK, the analysed Eurostat indicator is higher than the PIAAC indicator; while in Czechia and Belgium, the two indicators have similar levels. These findings suggest that the misreporting phenomenon may look different across populations.

The disadvantages of using direct assessments include the high costs and the limitations in the number of skills that can be measured. In the PIAAC, only basic skills related to literacy, numeracy, and problem-solving in technology-rich environments were subject to testing. But given that different job-specific skills are likely to be of interest to analysts and policy-makers, we expect that the direct assessment of skills will become more and more common. On the other hand, a potential alternative to relying on direct assessments is to use existing special methodological approaches for measuring sensitive behaviour as randomised-response (Böckenholt and van der Heijden 2007; Kirchner 2015) or item-count techniques (Holbrook and Krosnick 2010). Although the level of ICT skills does not seem to be a sensitive topic, the presented results suggest that it should be treated as sensitive. In addition, the application of the abovementioned methods to learn more about this phenomenon could be tested.

Finally, the importance of ICT skills should be stressed. The lack of an appropriate level of ICT skills can be considered from the perspective of the digital divide; a topic that has garnered considerable attention in the literature (e.g., van Deursen et al. 2011; van Deursen and van Dijk 2019; van Dijk 2006). While the concept of the digital divide initially referred to having physical access to computers and the internet, it has been extended to include possessing the right skills (digital skills, computer or media literacy), and to information inequality or information gaps. In the developed countries, the divides in material access to computers and other digital technologies are smaller than the divides in skills. Regardless of how large these gaps are, it is important to bear in mind that individuals who are unable to properly assess their skills may be less likely to believe that they need to improve their skills or seek professional guidance. Such an effect has been shown for financial literacy, with overconfident consumers being less likely to seek professional advice in some financial domains (Porto and Xiao 2016). Psychological experiments have revealed that an individual’s self-assessment of his/her knowledge increases after searching for information online, even about an unrelated topic (Fisher et al. 2015). It would be interesting to explore the computer and internet usage patterns of individuals who overreport their basic computer skills, such as the skills investigated in this paper.

Inequalities in ICT skills may be transferred to other life domains, and may have an impact on, or even reinforce, other types of inequalities, including immaterial, material, social, and educational gaps (van Dijk 2006; DiMaggio et al. 2001; Hargittai and Shafer 2006; Robinson et al. 2015; Selwyn 2006). Today, having access to information is critical, and it can shape an individual’s life chances in multiple ways. Public policies should be designed with the aim of ensuring that the number of actual or potential digital illiterates is as small as possible. But before individuals can acquire the right skills and physical access to ICT devices, they must be motivated to do so. At the same time, official statistical agencies should be able to correctly assess the level of ICT skills in the population, and public policy-makers should seek to mitigate existing and potential inequalities in these skills. We believe that the results of our analyses will be helpful in selecting the right direction for methodological developments in ICT skills measurement.