Teachers’ data literacy for learning analytics: a central predictor for digital data use in upper secondary schools

The use of educational data to inform pedagogical decisions has been explored in research related to schools. In this context, data literacy for teaching (Mandinach & Gummer, 2016) or pedagogical data literacy (Mandinach, 2012) is an important competence that affects teachers’ data-informed classroom practices and contributes to better student learning (Dunn et al., 2013; Kippers et al., 2018; Poortman & Schildkamp, 2016). According to a recent review, more research is needed on data literacy to integrate a situated view of educational data into everyday life with an emphasis on context-specific evaluations (Gebre, 2022). Even before the digital era, school teachers had access to a variety of educational data to inform their pedagogical decisions at the classroom level (e.g., student achievement data, student attendance data, and student products). Currently, the increasing use of digital learning environments and student information systems in schools enables the storing and analysis of additional types of educational data, and digital formats offer further opportunities to store, process, and display data for pedagogical decision making.

Students interact daily with learning management systems, digital learning applications, or digital communication platforms in their everyday practice, and this became even more pervasive during and after the COVID-19 pandemic. The analysis and use of digital educational data are of high relevance in educational research, and emerging fields, such as learning analytics and educational data mining, can enrich traditional approaches to understanding and optimizing the conditions for teaching and learning (Papamitsiou & Economides, 2014). However, new forms of collecting, analyzing, and using digital data in schools with digital platforms have just started to be explored (Kovanovic et al., 2021; Mandinach & Abrams, 2022; Sousa et al., 2021). Although learning analytics tools are increasingly available in school contexts (Kovanovic et al., 2021), research related to teachers’ uses of digital data is still in its infancy.

Previous research on teacher data literacy in schools has explored relevant factors that affect teachers’ data use (Mandinach & Gummer, 2016; Mandinach & Jimerson, 2016). The present study seeks to further investigate different factors that affect teachers’ digital data use across different digital platforms in schools, with reference to technology integration research. We conducted a survey study in upper secondary schools in Switzerland with the aim of enhancing the understanding of digital, data-informed teacher practices in schools and discussing the consequences of the future use of learning analytics in K–12 contexts.

The successful implementation of data-driven instruction and learning analytics tools in classrooms also depends on teacher practices. In a literature review, Datnow and Hubbard (2016) identified several factors affecting teachers’ effective use of student data, such as teachers’ beliefs, efficacy, and capacity to work with educational data (Bertrand & Marsh, 2015). Other studies have focused on the various forms of support given to teachers to use data effectively. For instance, access to technologies that provide data or relevant computer data systems, the available, structured time to reflect with data, and school leadership have been shown to influence teacher data use in schools (Rangel et al., 2016). A relatively high number of studies have focused on teacher data literacy with the aim of studying teachers’ engagement and the pedagogical uses of student data (Henderson & Corry, 2020). Mandinach & Gummer (2016) described teacher data literacy as “the ability to transform information into actionable instructional knowledge and practices by collecting, analyzing, and interpreting all types of data (assessment, school climate, behavioral, snapshot, etc.) to help determine instructional steps” (p. 367). This involves various factors, such as teacher beliefs, teacher pedagogical content knowledge, and experiences with data inquiry. Most studies on teacher data literacy conclude that a better framework for capacity building and relevant teacher training (Farley-Ripple & Buttram, 2015) is required to improve teacher data use.

The current digital transformation of schools has changed practices regarding the collection, management, analysis, and processing of educational data. According to Jarke and Breiter (2019) digital data

Compared to findings in previous research on teacher data literacy, the use of digital technologies in schools now enables the automatic collection and production of “digital trace data” (Breiter & Hepp, 2018) in real time. Teachers are considered the main educational stakeholders who are presented with these new forms of data, and this happens through access to data visualization tools, teacher dashboards, data technologies, and student information systems (Sampson et al., 2022). The rise of digital trace data and data technologies in education is expected to influence research and practices in learning analytics (Papamitsiou & Economides, 2014) in the school context (Kovanovic et al., 2021; Sousa et al., 2021). However, although empirical evidence on teachers’ use of traditional educational data has shown positive effects on student learning, there is a lack of evidence regarding the use of these new possibilities in using digital student data in schools.

The use of digital platforms that provide data displays to present real-time or more detailed insights into learning processes by using sophisticated statistical models (e.g., artificial intelligence techniques) to adapt to student needs and to provide personalized learning experiences leads to new skill demands for teachers (Celik et al., 2022; Howard et al., 2022; Mandinach & Abrams, 2022; Sousa et al., 2021; Sampson et al., 2022). Although teachers have more opportunities to use these new data technologies, there is still no systematic research on how these technologies are used effectively in practice. In particular, there is a lack of large-scale studies that seek to understand whether and why teachers engage with these data technologies in schools.

Some studies have explored teachers’ perceptions or use of digital data in school contexts on a small scale. For instance, Jover (2019) conducted interviews in Australian schools and discovered both positive and negative perceptions of teachers with regard to digital student data. Among the positive perceptions were that teachers are eager to work with digital data because they can adapt their teaching strategies to their students. They also felt satisfied when they saw visual forms of data and perceived the availability of data on digital platforms as easy. Among the negative teachers’ perceptions were that engagement with digital data could be a time-consuming process, the risk of overestimating quantitative data, and possible overload due to data deluge. This is in line with other empirical studies showing that teachers’ beliefs and previous experiences are related to teachers’ sense-making of data (Bertrand & Marsh, 2015).

Large-scale survey studies have evaluated analog data for teachers or assessment data without focusing on data collected on digital learning platforms. These studies have shown that teachers’ intentions to use data, teachers’ beliefs, and student outcomes are related to teacher data use (Keuning et al., 2019; Kippers et al., 2018; Pierce et al., 2013; Prenger & Schildkamp, 2018; Yan et al., 2021). Relevant theoretical frameworks were used to explain factors influencing teachers data use in schools including the theory of planned behaviour (TPB) (Ajzen, 1991) and technology acceptance model (TAM) (Davis, 1989). Prenger & Schildkamp (2018) used the TPB and showed that perceived control; the degree to which teachers have autonomy to make instructional changes based on data, their beliefs that data can lead to improvement instead of their experiences and their intentions to use data significantly predicted data use in schools. Schildkamp (2019) further explains that big data have just started to be explored for school improvement and teacher use but socio-technical and ethical challenges should be considered in school contexts. Mavroudi et al. (2021) conducted a survey study based on TAM to understand school teachers’ views about learning analytics. Explorative analysis showed that perceived usefulness of learning analytics applications was positive but teachers had moderate views with respect to their scepticism, willingness, and readiness about learning analytics in their school practice. The survey study by Hase et al. (2022) made the first attempt to identify factors that support the integration of digital learning platforms and learning analytics in teaching at a larger scale. The authors applied the TPB to demonstrate that teacher competencies, operationalized as perceived behavior control, relate to the use of learning analytics. However, although teacher competencies with data seem to influence the use of digital data, there is still a gap in research on identifying related factors that affect teacher use of learning analytics collected on digital platforms in K–12 on a larger scale (Sousa et al., 2021).

Given the current state of research, the available evidence suggests the need for a better investigation of teacher characteristics, such as their age, technological skills, and gender, and teacher factors, such as data literacy, professional routines, and pedagogical knowledge (Hase et al., 2022; Ifenthaler, 2021; Kovanovic et al., 2021; Leeuwen et al., 2019, 2021; Molenaar & Knoop-van Campen, 2018; Sousa et al., 2021). In addition, further investigation of the sociotechnical system and the school context that shapes teachers’ practices with regard to data use (e.g., their school environment and technological infrastructure) is needed (Datnow & Hubbard, 2016; Michos et al., 2018; Knauder & Koschmieder 2019; Sousa et al., 2021). As there are few dedicated studies on these contextual factors influencing the uptake of learning analytics in schools, it can be assumed that similar factors, such as those in general technology integration research, might play a central role. Beyond the availability of suitable hardware and software, teacher skills and teacher beliefs have been identified as core factors of technology integration in education (Davies & West, 2014; Niederhauser & Lindstrom, 2018; Tondeur et al., 2020). Therefore, research on learning analytics technologies in schools could be guided by models such as the “Will-Skill-Tool” model (WST model), which explains a large portion of the variance in the pedagogical uptake of technologies by teachers (Knezek et al., 2000; Petko, 2012; Petko & Prasse, 2018)

Teachers’ digital data use is an emerging topic due to the increasing use of digital technologies in schools. Although there is initial evidence showing that school teachers lack data literacy skills with regard to data use for informing their teaching, there is a limited number of studies that evaluate this at scale in K–12 education. In addition, few studies have explored relevant factors and types of digital platforms that are associated with more intensive use of digital student data for teaching purposes. Based on our theoretical grounding, models that explain factors of technology use by teachers have been scarcely applied in research on learning analytics in schools. For this reason, we conducted a survey study with N = 1059 school teachers in Switzerland based on the WST model that focuses on teachers’ beliefs, skills and availability of tools in schools. With respect to our first research question, our survey study shows that more than 50% of teachers in gymnasiums, technical, and vocational schools indicated having access to technologies that provide digital student data. Thus, there is an opportunity to use digital student data to better inform their teaching. However, only 30% of teachers reported using digital student data for their teaching and 26% of teachers felt confident in using digital student data for this purpose. These results align with previous studies showing that teachers’ data use for instruction varies across districts or schools (Cho & Wayman, 2014). These studies in schools highlighted the importance of accessing relevant computer data systems to use data in practice (Cho & Wayman, 2014; Schildkamp, 2019), however there is a lack of evidence with respect to digital data and learning analytics use by teachers.

In our second research question, we investigated whether teacher and contextual factors such as the different schools influence teachers’ use of digital data. We employed the WST model of technology integration to systematically investigate these factors that positively predict the pedagogical use of digital data by teachers considering that teachers are nested in schools (Rangel et al., 2016). Teacher data use was operationalized based on related items from research on school contexts (Wayman et al., 2016) and was adopted for the purposes of digital data. Our multilevel model initially showed a variance in teachers’ digital use between schools. This result might be explained by the type of schools participated in the study (e.g., gymnasiums, technical, and vocational schools) and possible different subject teachers that use digital data. Our theoretical assumption that the WST model will be relevant in the context of digital data use in schools is confirmed. In particular, the model showed that teachers’ positive beliefs about digital learning technologies (will), their data literacy (skill), and access to relevant data technologies (tool) positively affected teachers’ digital data use to inform instruction. In particular, teacher data literacy was accessed as the central predictor in our model in upper secondary schools and this aligns with previous research on teacher data literacy in schools (Schildkamp, 2019). The additional factors added to the model showed that the frequency of using digital devices by students in teachers’ lessons affected teachers’ digital data use to inform instruction. However, the frequency of using digital devices by teachers did not affect digital data use because in our sample most teachers responded using a digital device in almost every lesson. Teachers’ characteristics, such as their teaching years and age, slightly influenced data use, but their gender seemed to be an important factor in digital data use.

Our results confirm related studies that have investigated factors such as access to data management systems, teachers’ learning on how to use data, and teachers’ beliefs (Rangel et al., 2016; Datnow & Hubbard, 2016). Our study added that research on teachers’ use of digital data can be guided by the research on technology integration with the WST model. Moreover, relevant studies have found that teachers’ sense-making of educational data is shaped by the context, district, networks, and school leadership (Rangel et al., 2016), and our study shows that differences in teachers’ data use are attributed to the differences in the context of schools. Datnow and Hubbard (2016) offered a model that shows that teachers’ attitudes and beliefs are important in actual data use. When teachers are provided with opportunities to discuss data-related practices with colleagues, they perceive higher control. Howard et al. (2022) highlighted a growing interest in using digital data in schools, which aligns with the adoption of digital technologies and learning management systems (Baker et al., 2020).

In our third research question we focused on the availability of digital platforms available in schools. Our analysis of the different platforms within our sample shows that schools often use cloud-based platforms for collaboration, reading/writing, learning management systems, videoconferencing, and platforms that provide learning analytics. The results show significant differences in teachers’ use of digital data and data literacy when teachers had access to cloud-based platforms for collaboration compared to when they lacked access to these platforms, suggesting that collaborative platforms could provide more opportunities for the pedagogical use of digital data in schools compared to the other platforms’ types included in our analysis. Surprisingly, access to LMSs did not show any differences in the use of digital data. Although the surveyed teachers seemed to have good access to different technologies that provide learning analytics, they reported low levels of data use and data literacy. This could potentially prompt the question of whether the teachers were aware of the data analytics functionalities of the different types of platforms that were at their disposal in their schools. Further studies may evaluate platforms based on the specific learning analytics or personalized learning functions provided to teachers and school administrators to better understand the relationship between access to data technologies, data literacy, and use.

Based on our results, there seems to be a need for better capacity building in teachers’ use of digital data (Farley-Ripple & Buttram, 2015), and this is also relevant to digital platforms that provide digital student data. Capacity building might facilitate teachers’ understanding of the different types of digital data (e.g., assessment, attendance, and student progress) as well as the possible different uses to inform their teaching in schools (Howard et al., 2022). Jimerson & Wayman (2015) argued that “Training for data use often is synchronous with technology training” (p. 36), and therefore, a connection with training on digital technologies for teaching is needed. Finally, specific plans for the development of data-informed practices in schools, as recommended in previous research (Jimerson & Wayman, 2015), remain pertinent. Our results add more evidence to large-scale studies on teacher data use (Keuning et al., 2019; Kippers et al., 2018; Pierce et al., 2013; Prenger & Schildkamp, 2018; Yan et al., 2021) and expand the findings in the case of digital student data or learning analytics in the K–12 context (Hase et al., 2022; Mavroudi et al., 2021) showing that different school contexts and digital platform types can influence teachers’ use of learning analytics (Sousa et al., 2021).

Our study shows that teachers differ in the use of digital data, and this is influenced by teachers’ positive beliefs towards digital technologies (will), the self-perceived competency in using these platforms (skill), the available technologies (tool) and the general frequency of using digital devices in lessons by students. Thus, we were able to confirm and expand the application of the WST to teachers` data use. Our results confirm previous research on teacher data literacy in schools and provide further evidence regarding the digital data literacy of teachers. This is also one of the first studies that provide empirical evidence with regard to teacher data literacy and learning analytics in school contexts. Our study has certain limitations, including the fact that we measured teachers’ beliefs regarding digital technologies in general instead of digital data in particular. In addition, due to the constraints of survey length, we were only able to use single items to measure teacher data use, access to technologies, and teacher data literacy. Further studies might construct scales related to teacher data use with items relevant to teacher data literacy. In addition, we did not restrict our evaluation to specific types of digital data (e.g., assessment data and participation data); instead, we relied on teacher responses regarding various types of digital data provided on digital platforms. Future studies might focus on a specific type or different category of digital data that might inform teaching. However, our main purpose was to understand teachers’ perceptions and factors of digital data use on a larger scale in schools. More detailed research is needed regarding various factors, such as the school context, access to available technologies, the frequency of using digital devices, teachers’ positive beliefs toward digital technologies, and teacher competency with digital data. Finally, teacher training in the use of digital data provided in learning analytics tools is recommended to develop teacher data literacy in schools.