The effects of a data use intervention on educators’ satisfaction and data literacy

To further specify what kind of data literacy skills are needed by educators to use data in their schools, it is necessary to specify how schools can effectively use data. As presented in Fig. 1, data use involves an interpretative process, in which data have to be accessed, collected, and analyzed to be turned into information, and must be combined with understanding and expertise to become meaningful and useful for actions (Coburn and Turner 2011; Marsh 2012). Based on Marsh (2012, p. 4), and incorporating relevant characteristics of other data use models and frameworks (Coburn and Turner 2011; Ikemoto and Marsh 2007; Schildkamp and Lai (2013); Mandinach et al. 2008; Schildkamp and Kuiper 2010; Supovitz 2010), a data team theory of action (Fig. 1) was developed (Schildkamp and Poortman 2015). In this framework (Fig. 1), the interaction between data and people, in a certain context, results in decisions with regard to what action to take.

For implementing data use in schools, the data use intervention in the form of data teams was designed. This data use intervention aims at enhancing educators’ data literacy skills about data use by giving educators a structured approach containing eight steps (see Fig. 2). This eight-step approach supports the team members to solve a problem emerging in their own school context by using (qualitative and quantitative) data, and it supports the active involvement of members (Bryk et al. 2015). Engaging educators in conversations regarding educational problems within their daily practice by using data creates powerful professional development opportunities, builds collegiality, and helps building professional relationships (Coburn and Turner 2011; Halverson 2010; Piety 2013; Brocato et al. 2014). The main objective of the data use intervention is professional development regarding data use by collaboratively solving a realistic problem defined and owned by the data team members.

In this study, data teams consisted of 4–6 teachers, 1–2 team leaders/school leadership team members, and if available in the school, also an internal data expert. These team members work together to solve an urgent educational problem in their own school context. Working according to the data team intervention gives educators within schools ownership about the process and makes educators active agents for data use (Bryk et al. 2015).

Data teams work according to a cyclic and iterative approach of the eight steps leading to the implementation of improvement measures based on data analysis. There is a general consensus about the steps that are important for effective data use in schools, albeit the steps vary across publications (see e.g., Boudett et al. 2005; Earl and Katz 2006; Marsh 2012). The eight steps in the data team intervention were inspired by existing data use manuals (e.g., Earl and Katz 2006; Boudett and Steele 2007). Furthermore, the intervention incorporated in the key elements of the data use theory of action (Fig. 1).

The eight step approach starts with a purpose by defining the problem (step 1). This means that the data team members brainstorm about educational problems within their school and goals they want to focus their efforts on. Making the problem specific is an essential step, because it gives a clear direction to data team members (Bryk et al. 2015). Next, data teams formulate concrete and measurable hypotheses about possible causes of the problem (step 2), for example, about possible causes for low examination results in mathematics. Subsequently, team members collect data to investigate the hypotheses (step 3). Qualitative and quantitative data collection methods can be employed for collecting, e.g., assessment data, examination results, or student voice data. In the next step, data team members check the quality of the data (step 4) regarding their reliability and validity. For example, members check to which degree the instrument they used for data collection measured what it claims to measure. Next, members analyze the data (step 5) to be able to verify or reject the hypotheses under investigation. This may involve statistical procedures (e.g., descriptive analyses and t tests) or the analysis of qualitative data (e.g., coding and summarizing data). Subsequently, data team members interpret the analysis of the data and draw conclusions based on the analysis (step 6) implying data are to be transformed into information. Together with the participants’ expertise and skills, this information is turned into knowledge. This step also implies that if the tested hypotheses turn out to be false, new hypotheses have to be formulated and tested. In that case, teams have to go back to step 2. Next, the data team takes action by implementing improvement measures (step 7). Finally, the team evaluates the implemented measures (step 8) by establishing the outcomes. Here, the team determines if the improvement measures were effective and if the goals were met. The eight steps are completed only if the goals have been accomplished.

The data teams are coached in following the data use approach by an external data coach for a period of one-and-a-half years. The external data coach joins each data team for 90 minutes every 3 to 4 weeks. It is the external data coach’s task to monitor the process of the data team and give just-in-time support.

This intervention meets many criteria of effective professional development as found by researchers in the field of professional development. These include collaboration in a professional learning community, shared vision and goals, related to daily practice, active participation, leadership, structure, and support (e.g., Borko 2004; Desimone 2009; Guskey 2000; Jimerson and Wayman 2012; Lomos et al. 2011; Stoll et al. 2006; Vescio et al. 2008).

According to Guskey (2000), “professional development is defined as those processes and activities designed to enhance the professional knowledge, skills, and attitudes so that they might, in turn, improve the learning of students” (p. 16). For being able to use data effectively, educators need to develop their data use attitude and data literacy skills (Marsh 2012). Data literacy skills are the skills and knowledge that are needed to effectively use data in schools (Gummer and Mandinach 2015). Developing a positive attitude toward data use, or “buy-in/belief in data” (Schildkamp and Kuiper 2010), is necessary for the implementation of data use in schools (Datnow et al. 2007). Having or developing a positive attitude toward data use in this study means that participants recognize the added value of analyzing and using data and of having evidence for the claims they make regarding problems in their school.

Taking into account the steps that educators need to complete in order to effectively use data in their schools as presented in the data use theory of action (see Fig. 1), data literacy skills are, for example, knowledge and skills about accessing, collecting, and analyzing data for investigating hypotheses.

A prerequisite for being able to acquire data literacy skills in an intervention is the satisfaction of the participants about the intervention (Desimone 2009; Guskey 2000), e.g., that participants are satisfied about the material that is provided, the extra courses regarding data analysis that are offered by the university, and the support as provided by the external data coach during the data team meetings. Testing satisfaction about the intervention is important, because the higher the satisfaction of participating educators with the intervention, the more likely they will obtain data literacy skills, as provided in the intervention, which they can use in practice in order to improve education (Nir and Bogler 2008).

Consequently, in this paper, we focus on the effects of the data team intervention on educator satisfaction and on educator learning outcomes (data literacy skills and attitude) with regard to data use. To evaluate the effects of participating in a data use intervention on educators’ professional development regarding data use, our two research questions are as follows:

(RQ2) To what extent have educators’ data literacy skills and attitudes improved after participating in the data use intervention?

This study took place in the Netherlands. The Dutch Inspectorate holds schools accountable for the education they provide. Schools can use several data sources in order to improve their education, including, for example, school inspection data; school self-evaluation data; data on intakes, transfers, and dropouts; final examination results; assessment results; and student and parent questionnaire data (Schildkamp and Kuiper 2010).

The policy emphasis for data use has increased significantly in the Netherlands in recent years, as is the case internationally (Schildkamp et al. 2014). The Dutch School Inspectorate is increasingly holding schools accountable for using data to improve their quality (Verbeek and Odenthal 2014). It is the aim of the Ministry of Education that by 2018, at least 90% of primary and secondary education schools in the Netherlands apply data use. In 2010, only 20% of the Dutch secondary schools already applied data use (Dutch Inspectorate 2011).

This project is part of a cooperation with one of the largest Dutch school boards in secondary education. Ten schools from this school board voluntarily signed up for participation with a data team. The 42 schools of the school board that did not participate in a data team formed our comparison group (see Table 1). These 42 schools are a naturally more comparable group for the intervention group than schools from other boards, because these schools operate under one board with the same vision and general policy regarding data use.

Furthermore, we analyzed whether the two groups were comparable by calculating chi-square tests regarding the two demographic variables: the gender of respondents and the subject area of respondents (i.e., languages, science and mathematics, or other subjects, for example, art, drama, PE, and sport) (Table 2) (Field 2013). Chi-square tests of independence were calculated comparing the frequency of the gender of respondents and the affiliation of respondents (affiliation in terms of working in a data team school or school belonging to the comparison group) and comparing the frequency of the subject area of respondents and the affiliation of respondents. A non-significant interaction was found for gender (X ²(1) = 1.043, p = 0.307) and subject area (X ²(2) = 0.878, p = 0.645). Therefore, we can conclude that the groups are comparable with regard to these aspects.

The problems these data teams were working on included, for example, the high retention rate in the fourth grade of senior secondary education (providing access to polytechnics) and the disappointing final examination results for English. Three data teams were selected for a qualitative case study during one-and-a-half school years, in which they were supported by the external data coach.

In order to select the three case study schools at the start of the project, first, a cluster analysis was run on all schools with among them nine schools having a data team, each responding to items about data use actions in school (i.e., data use for instruction, school development, and accountability; for more information, see the section on the data use questionnaire and Ebbeler et al. 2016). A hierarchical cluster analysis using Ward’s method (Burns and Burns 2008; Saunders 1994) produced five clusters, between which the means of data use actions were significantly different. The clusters ranged from low to high means of reported data use in the schools.

Furthermore, the selection was based on teams being comparable regarding the presence of at least one school leader, an internal data expert, and at least three teachers during the meetings. This resulted in a selection of three case studies from three different clusters, ranging from low to high average scores per school for data use (see Table 3 for more information).

We used a combination of different instruments for triangulation purposes in order to adequately answer both research questions. In Table 4, we present all of the research instruments related to the research questions, as well as the data analysis conducted to answer the research questions.

The satisfaction questionnaire results show that data team participants were, on average, satisfied to very satisfied about the materials (N = 55; M = 4.14; SD = 0.62) used in the data use intervention. The interviews with data team members showed similar results. In school A, for example, an interviewee stated that he appreciated the material as provided by the external data coach during the data team meetings, and also recognized that the material was updated during the intervention.

Inspection of the gain scores from pre-test to posttest survey results indicate that at the end of the intervention period, mean scores for the data literacy skills and attitude increased more for teachers in data team schools (M = 0.10; SE SD = 0.0412) than for teachers in the comparison group schools (M = −0.06, SE SD = 0.0526). The results of the independent samples t test show that these differences were statistically significant t(40) = −1.747, p = 0.04 (see Table 8). The effect size d = 0.6 represents a medium effect (Cohen 1992; Field 2013).

Regarding data literacy skills as measured by the knowledge test, data team members scored higher on the posttest (M = 10.4; SD = 3.05) than on the pre-test (M = 9.4, SD = 3.16). This difference, −1.04, BCa 95% CI [−1.82, −0.27], was significant t(35) = −2.72, p = 0.005. The effect size d = 0.32 represents a small to medium effect (Cohen 1992; Field 2013).

In the interviews, respondents from school A reported that the data team members have learned how to set up a good questionnaire by (re-)designing a questionnaire through exchanging feedback on that questionnaire. Team members also reported they learned what a scatterplot is. In school C, one of the respondents explained that they have learned how important data are to better formulate goals when trying to improve something in school.

Another respondent in school B gave a specific example of improving his data literacy skills: “I never worked with Excel for conducting statistical analyses. That is what I have learned: to calculate p values in Excel. And to be able to do it is really good” (interviewee school B). Also, data team members reported that they learned, e.g., how to calculate correlations in Excel (interviewee school B), that working with the data management system can be quite difficult (interviewee school B), and they learned about qualitative data analysis in the data analysis course (interviewee school C).

The interviews indicated that data team members’ data literacy skills have improved. However, the interviews also provide some explanation for why the effect size from the knowledge test was only small to medium. Several respondents stated that they did not learn anything really new with regard to data analysis (interviewees in schools A, B, and C). However, it is important to realize that respondents that gave this statement were mostly mathematicians, teachers with a background in sciences, quality care managers, or teachers who (recently) graduated from university with a master’s degree.

Also, an interviewee reported being more critical toward colleagues who are jumping to conclusions without having evidence. If colleagues gave solutions for a problem or assumed that they knew the cause for the problem, some data team members reported that they would ask these colleagues to prove these preconceptions (interviewee in school B).

However, there were also some critical remarks. In school B, one participant was not completely open to using data and did not fully accept the outcome of the data analysis. Also, in school A, there were some doubts about using data because the eight-step procedure had not been finished at the time of the interviews.

During the evaluation with the external data coach, data teams made some critical remarks as well, e.g., that the team will not continue working according to the eight-step approach (school A), and it became clear that not everyone on the team was convinced of the importance of using data for proving assumptions (school B).

This study shows that structured interventions such as this data use intervention can provide an important contribution to improving data use capacity building in schools. Crucial is that these data literacy skills are built using data from the teachers’ own context and that teachers collaboratively use data. Unlike several other data use interventions (e.g., Carlson et al. 2011; McNaughton et al. 2012), this data use intervention starts with a problem the school chooses, and not with a general subject, such as literacy or mathematics.

Moreover, it is in the interaction within the team, between team members, and between the coach and the team members that based on data the improvement process starts. Through collaborative data discussions, taken for granted assumptions are disproven, myths are dispelled, actual causes of educational problems are found, and learning and improvement starts. This is what Earl and Timperley (2009) refer to as learning conversations. Some interventions use steps such as “digging into data” and “creating data overviews” (e.g., Boudett et al. 2005), and then move into the hypotheses (why) phase. The data teams move into the hypothesis phase sooner, to start with the learning conversations as soon as possible, to create a sense of ownership, and eagerness to learn.

Moreover, a structured step by step approach as used in this intervention is also important. The intervention has similar steps as the Plan Do Check Act cycle of Deming, and activities described in other data use projects and books (e.g., Boudett et al. 2005; Boudett and Steele 2007; Earl and Katz 2006). Data use with the goal of school improvement will always have similar components, including determining a purpose, data collection, from data to information to knowledge, to action, and evaluation (see also our data use theory of action). The results of this study show how important it is to make sure these steps are made explicit and concrete (and even then, it is still a challenge to develop data literacy).

Furthermore, the results of this study point to the importance of university–school partnerships. A coach from the university guided the team through all these steps. The data team coach supported the team, but the team also provided the data team coach and the university researchers with feedback. Based on the interaction with and feedback from the data teams, the data use intervention has been improved, for example, by adding more examples in the manual and by adding work sheets to the manual.

To summarize, important aspects for data use professional development interventions include the following:

However, there are also challenges associated with these types of data use interventions. This study also shows that professional development requires time. Firstly, time for developing data literacy skills, and secondly, time for the required attitude shift. The attitude shift is needed, because educators need to feel the urge to use data. This shift cannot be created within a few meetings, but grows over time. Therefore, when developing data literacy, it is important to realize that these skills cannot be developed in just a couple of workshops, but that a long-term collaborative approach at the school site is much more likely to lead to the desired results.

The key challenge for practitioners is to share their data literacy skills with their other colleagues in the school and not only within a small team. The way in which practitioners shape the process of knowledge sharing regarding data use will influence the sustainability of the intervention, and thus, also whether and how schools will use data for school improvement in the future. Therefore, we need further research into how data teams will use and share their new data literacy skills in practice.

Using and sharing data also requires a certain openness in the school and a level of trust. Only if educators are open and willing to use data to reflect on their own classroom practices, we can also expect improvements based on data within classrooms. This is especially important in low-performing schools, where educators often lack basic teaching skills (Mintrop and Sunderman 2013).

Some limitations of this study along with suggestions for further research should be noted. Future research should take into account that the knowledge test used in this study was only administrated within the intervention group and not in a comparison group. We tried to create a complete picture by triangulating the knowledge test results, with interviews and questionnaire results.

In this study, we showed that the participants have learned new data literacy skills. This study did not only use perception data in demonstrating these results, but we went beyond perception data by using a knowledge test to measure the effects on the knowledge and skills of participants. However, as other studies have shown, positive results at the learning level do not automatically lead to effects on student achievement (e.g., see Jacob et al. 2014). Future research should focus on the question of whether the participants actually apply their new data literacy skills in practice. Only if the new data literacy skills are applied in practice can improved student achievement, which is the ultimate goal of professional development, be reached (Kirkpatrick 1996; Guskey 2000). For example, some of the schools reported that their data team had managed to find the causes of their problem and started to implement measures to improve their student achievement. Further research is needed in order to assess whether these measures have had an impact on student achievement. From other studies, we know that this is possible, as these (quasi) experimental studies have found an effect of data use interventions on student achievement (e.g., Carlson et al. 2011; McNaughton et al. 2012).

In addition, as one of the respondents stated in an interview, the question is what happens after the external support ends. The aim is that teams are able to continue with data use after the support has ended. Therefore, not only short-term research into the effects but also research into long-term effects, i.e., the sustainability of data teams, is important.