It has long been recognized that early childhood programs of high quality promote children’s school readiness and longer-term educational success (Camilli et al.
2010; Karoly et al.
2005). These identified gains have also been found to lead to economic and social benefits in adulthood (Cannon et al.
2017; Reynolds and Temple
2008). Since 4 in 5 children in the USA between the ages of 3 and 5 years old participate in center-based education and care for at least part of the day (U. S. Department of Education
2017), ensuring that programs are high quality is a major priority. However, wide variability in quality is the norm, especially for children from low-income families.
In center-/classroom-based early childhood learning environments, specific aspects of classroom practice have been linked to children’s social and academic outcomes. For example, teachers’ levels of education (minimum bachelor’s degree) are associated with children’s development of social competence (Mashburn et al.
2008) and higher receptive language skills (Burchinal, Cryer, Clifford, & Howes,
2002). Further, high-quality instructional practices and teacher-child interactions in early childhood programming have replicated linkages to children’s academic and social development (Gosse et al.
2014; Mashburn et al.
2008; Wasik and Hindman
2014). Both quality factors are important in supporting children’s gains, yet structural features may be necessary for establishing a supportive context for learning. Below we summarize key learning experiences that are promoted by high quality classroom environments and their relations with school achievement.
4 Methods
4.1 Sample and setting
Study participants are part of the Midwest Child-Parent Center (MCPC) Expansion Project, an evidence-based prekindergarten (PreK)-3rd grade school reform model implemented beginning in 2012–2013 in four school districts (Reynolds et al.
2014, Reynolds et al.
2016a,
b). In order of size, they are the Chicago Public Schools, Saint Paul Public Schools, Evanston-Skokie District 65, and McLean County Unit District 5. MCPC is funded by an Investing in Innovation (i3) grant from the U.S. Department of Education. The 5-year intervention provides comprehensive family and school support services to a cohort of children from PreK to 3rd grade (Reynolds et al.
2016a,
b,
2017). The six core elements are collaborative leadership, effective learning experiences, aligned curriculum, professional development, parent involvement and engagement, and continuity and stability. School, classroom, and teacher services include an aligned professional development/coaching model, leadership support, classroom aides, and vertically and horizontally aligned curricula. A total of 98% of the PreK teachers had at least Bachelor’s degrees with an average of eight years of teaching experience.
As shown in Table
1, the original sample is a PreK cohort of 3535 students in 46 schools (2323 program and 1212 comparison-group children). The comparison group enrolled in the usual district preschool programs in schools matched on propensity scores (student demographic characteristics and 3rd grade test scores). Although the present study does not assess the intervention, this school context provides a description of the sample selection.
Table 1Midwest Child-Parent Center (MCPC) school, classroom, and student sample sizes
Number of schools | 46 | 33 | 24 | – |
Program, comparison | 25, 21 | 25, 8 | 16, 8 | – |
Number of classrooms | 116 | 72 | 54 | – |
Program, comparison | 88, 28 | 64, 8 | 46, 8 | – |
Number of students | 3535 | 2232 | 1358 | 70%, 70% |
Program, comparison | 2323, 1212 | 1950, 282 | 1134, 224 | 68%, 81% |
Chicago schools | 30 | 24 | 24 | – |
Classrooms and students | 86, 2630 | 54, 1358 | 54, 1358 | 60%, 61% |
Outside Chicago schools | 16 | 9 | N/A | – |
Classrooms and students | 30, 905 | 18, 874 | N/A | 89%, 85% |
Two samples for the current study were defined. The CLAC study sample included 72 (out of 116) randomly selected classrooms from program (
n = 64) and comparison schools (
n = 8; Chicago only). One or two classrooms were selected from each school depending on the number of PreK rooms. Accounting for morning and afternoon sessions in most classrooms, this included a total of 2232 students (see Table
1). Because Chicago is over 70% of the total sample, the validity sample was restricted to this district. Included were 54 classrooms (24 schools) with 1358 enrolled students, 60% of whom were 4-year-olds. All districts schools use the Teaching Strategies Gold Assessment System (TS-Gold; Heroman et al.
2010), a standardized performance assessment of school readiness skills. Table
2 shows the demographic characteristics of the validity sample.
Table 2Student characteristic and covariate sample sizes, means, standard deviations, and response ranges
Gender (Female) | .52 | (0.50) | 0–1 |
Black (African-American) | .68 | (0.47) | 0–1 |
Hispanic | .31 | (0.46) | 0–1 |
Special education placement | .07 | (0.26) | 0–1 |
Age in months | 48.25 | (6.48) | 35.35–58.84 |
Eligible for subsidized lunch program | .86 | (0.35) | 0–1 |
Fall assessment was after October (1 = yes) | .41 | (.49) | 0–1 |
Baseline TS-Gold math skills | 22.44 | (8.63) | 0–56 |
Baseline TS-Gold literacy | 33.35 | (15.50) | 0–92 |
Baseline TS-Gold socio-emotional | 39.91 | (12.86) | 0–81 |
Baseline TS-Gold language skills | 28.16 | (7.76) | 1.5–54 |
Baseline total score (all subscales) | 190.53 | (58.77) | 10.45–386 |
4.2 Classroom Learning Activities Checklist
As part of the study, classroom observations were conducted using the Classroom Learning Activities Checklist (CLAC), an internally created assessment that captured the nature and quality of student task orientation and the classroom practices that support it. Roughly one-half of the prekindergarten classrooms in each of the implementation sites and one classroom from each control site were randomly sampled.
The assessment tool was designed to be consistent with principles of effective learning environments described in the introduction and included content on engaged instruction and self-regulation, an enriching classroom climate, task-oriented goals and experiences, and active learning and child-initiated activities (Graue et al.
2004). These principles and foci are key elements in the CPC program and other effective interventions leading to beneficial long-term effects (Ramey and Ramey
1998; Reynolds et al.
2017; Reynolds and Temple
2019).
The CLAC is organized into 4 theoretically constructed domains: (a) items one through six inquire about observed student task-oriented behaviors; (b) items seven through 17 measure the provision and facilitation of learning activities that support task orientation; (c) items 18a–c, 19, and 20 assess how instructional time is spent; and (d) items 21–23 measure the presence and absence of student misbehavior. Each of these items is coded on a 1–5 Likert scale (1 = strongly disagree/never/none, 2 = disagree/rarely/few, 3 = neutral/sometimes/some, 4 = agree/most of the time/many, 5 = strongly agrees/always/nearly all) and has descriptions of each of the scores in a scoring rubric. Finally, item 26 (CLAC26) rates the overall level of task orientation in the classrooms. Assessors incorporate the four constructs into a single 1–5 score: 1 = very low, 2 = moderately low, 3 = somewhat, 4 = moderately high, 5 = very high. See Table
4 for a list of CLAC items 1–23 and item 26 (Table
9 for a complete list).
4.5 Analyses
A series of descriptive statistics were examined to explore the dimensionality of the CLAC. Exploratory factor analysis (principal components analysis) was conducted using SPSS version 14 and was used to (a) identify the dimensionality of the task orientation via reviewing the variance structure and (b) create factor scores that describe characteristics of task orientation for later regression analyses. The variables constructed from factor analysis were used in later predictive validity evidence analyses. Given that the CLAC tool was newly developed to measure task orientation and related attributes of quality, an exploratory analysis to identify the number of dimension using principal components analysis was preferred over a confirmatory factor analytic approach.
To assess the validity of CLAC scores in predicting later children’s learning, probit and multiple linear regression were used using STATA version 13. Linear regression analyses measured the relation of CLAC predictor variables to continuous outcome variables—children’s TS-Gold scores in language, literacy, math, socio-emotional, and total sum scores. To capture the potential impact on a minimum threshold of necessary learning, probit regression analyzes dichotomized outcomes of children’s proficiency where 1 = met national scores and 0 = did not meet nationally normed averages. Similarly, the relations among the CLAC variables with covariates to these binary scores were used to predict children’s language, literacy, math, and socio-emotional proficiency scores.
Regression coefficients in each model were used as indicators of the strength of relation between predictor variables (or covariate) and the outcome measure. Coefficients, either negative or positive, with a p value below .05 were considered significantly associated with TS-Gold.
Instead of removing incomplete observations (where one or two time points were missing) and decreasing power from lower sample sizes (Nakagawa and Freckleton
2008), regression models were analyzed using an imputed dataset. Multiple imputation of missing data using an EM algorithm was used to generate maximum likelihood estimates. This imputation method is often considered superior to other procedures that handle missing data (Buhi et al.
2008; Cox et al.
2014) while maximizing the available sample. EM algorithms provide excellent parameter estimates that are close to the population average (Graham
2009). Outcome and demographic variables, including fall baseline performance scores, assessment date, age, race, special education status, free lunch eligibility, gender, proficiency in three or more domains, and a school-level reading achievement, were included in the algorithm to produce missing case parameter estimates (means, variances, and co-variances).
5 Results
The purpose of this study was to explore the psychometric properties of the Classroom Learning Activities Checklist, including its internal design and structure, reliability, and validity evidence.
5.1 Research question 1: what is the construct validity evidence and dimensionality of the tool?
5.2 Research question 2: what is the predictive validity of the CLAC measure (its overall and two factor scores) to children’s learning using TS-Gold?
Three questions were addressed: to what extent can CLAC’s (a) instructional responsiveness, (b) student engagement, and (c) overall task orientation variables uniquely predict student’s TS-Gold scores, above and beyond a set of potential explanatory variables?
Each CLAC factor, along with covariates, was used as a predictor in linear and probit regression models for the TS-Gold outcome measures: Continuous and dichotomous measures of language, literacy, math, and socio-emotional were evaluated. A total continuous TS-Gold measure was created by compiling subscale scores. A proficiency total score was created by dichotomizing the total subscale score. Using a priori alpha level of .05, CLAC variables with coefficients that fell below this threshold were considered predictors of the student outcomes. Further, effect sizes were used to interpret the strength of these relations.
The models accounted for nesting by clustering the standard error at the classroom level and included the following covariates: baseline performance, assessment date, gender, race, ethnicity, special education status, age, and free lunch eligibility. There are several statistically significant relations between the covariates and TS-Gold outcomes (see Table
7). The dichotomized baseline achievement scores often perfectly predicted proficient outcome measures; consequently, the continuous baseline measures were used in the models. Multiple imputation via the expectation-maximization algorithm was used to fill in missing baseline and end-of-year scores after verification that values were missing at random. This approach increased power and efficiency in estimation.
Table 7Correlation matrix of covariates and spring outcome scores
1. Gender | – | | | | | | | | | | |
2. Race | − .01 | – | | | | | | | | | |
3. Hispanic | .01 | − .91 | – | | | | | | | | |
4. Special ed | − .17 | − .14 | .12 | – | | | | | | | |
5. Age | .001 | − .06 | .08 | − .001 | – | | | | | | |
6. Free lunch | .04 | .19 | − .15 | − .04 | .07 | – | | | | | |
7. Assess date | .04 | − .11 | .11 | − .01 | .10 | − .001 | – | | | | |
8. Language | .12 | .07 | − .07 | − .27 | .62 | .001 | .16 | – | | | |
9. Literacy | .08 | .12 | − .11 | − .20 | .65 | .02 | .15 | .87 | – | | |
10. Math | .07 | .07 | − .07 | − .22 | .65 | − .02 | .17 | .87 | .93 | – | |
11. Socio-emotional | .13 | − .03 | .03 | − .21 | .62 | − .004 | .18 | .90 | .82 | .83 | – |
12. Total Score | .10 | .05 | -.05 | -.22 | .67 | .007 | .17 | .96 | .95 | .94 | .94 |
Table
8 presents predictor variables’ model coefficients on each outcome measure along with each models’ effect size and
R2/pseudo
R2. Across all models, the total amount of variance accounted for in the models was quite high for the measures using the continuous predictors: Between 71 and 77% of the variance were accounted for by the individual models, i.e.,
R2 = .708,
p < .001,
R2 = .770,
p < .001. See Appendices
11,
12, and
13 for complete regression models with covariates for overall task orientation, instructional responsiveness factor, and student engagement factor, respectively.
Table 8Marginal effects and effect sizes from probit and linear regression model predicting year-end learning with CLAC scores
Factor 1 | 0.11** | .12 | 0.02* | .01 | 0.16* | .09 | 0.02 | .002 | 0.03 | .03 | 0.01 | .002 | 0.04 | .05 | 0.01 | .06 | 0.39 | .07 | 0.01 | .003 |
Factor 2 | 0.17** | .10 | 0.03* | .01 | 0.30* | .10 | 0.04* | .01 | 0.13 | .06 | 0.03† | .01 | 0.11* | .08 | 0.01† | .002 | 0.87* | .08 | 0.04* | .01 |
Overall task orientation | 1.53** | .34 | 0.24† | .05 | 2.47* | .30 | 0.26 | .04 | 0.39 | .07 | 0.07 | .02 | 0.62 | .18 | 0.08 | .01 | 5.97† | .21 | 0.13 | .03 |
6 Discussion
The purpose of this study was to assess for the first time the utility of the Classroom Learning Activities Checklist (CLAC) in measuring classroom task orientation and engagement in learning. The dimensionality and predictive validity were the main foci. The first research question focused on dimensionality. In analyzing the items associated with both factors, two subconstructs emerged: items that reflect classroom supports (instructional responsiveness, associated with factor 1) and observed reception to classroom activities (student engagement, associated with factor 2). The two CLAC factors reinforce existing literature that indicates different kinds of interactions (e.g., individualized attention and child engaging strategies) promote learning for young children (e.g., Burchinal et al.
2008; NICHD Early Child Care Research Network
2002). Specifically, responsive instruction with individualized support for children with low levels of self-regulation is associated with greater self-regulation gains (Connor et al.
2010). Student engagement, as previously examined in the literature review has also been linked to self-regulation (Fantuzzo et al.
2004; Williford et al.
2013). Additionally, these independent yet correlated CLAC factors support the bidirectional relation students have with their teachers and environments. Finally, our 2-factor structure is conceptually consistent with the CLASS domains of emotional support, classroom organization, and instructional support. Although we did not identify a third dimension, the CLAC tool was developed to assess a narrower range of classroom experiences and interactions consistent with the MCPC intervention theory.
The second research question examined CLAC’s predictive validity on children’s learning at the end of the PreK year. Without evidence of the measure’s ability to connect to child-level outcomes, our ability to further explore its potential refinement, scalability, and broader dissemination is limited. After controlling for covariates, each of the three predictors, overall task orientation (CLAC26), factor 1 (instructional responsiveness), and factor 2 (student engagement) significantly predicted year-end learning. The student engagement factor appeared to be the best predictor of children’s learning: seven of the 10 models with factor 2 were associated with significantly higher TS-Gold scores.
While it was encouraging to see evidence of the CLAC’s relation to aspects of student’s math and literacy learning, it was unexpected to not see differential impacts of the CLAC factor scores. Further predictive validity research is warranted: these analyses included data from one large Midwestern district using a single assessment, TS-Gold. The validity evidence of these findings would be bolstered by examining differing subpopulations and, more importantly, using additional standardized assessment tools with demonstrated validity.
While there was variability across the items, subscales, and factors, of concern is the consistent restricted response range of the individual CLAC items. After this observation round was conducted in the PreK year and its data analyzed, a number of measures were taken to address the lack of variability. First, a 7-point scale was piloted both independently and in conjunction with the 5-point scale. Observers reported anecdotally that the 7-point measure provided more variation in ratings. The scoring rubric was also revisited, and clarifying text was added where inconsistencies existed. These measures were taken to better distinguish values while maintaining the same scoring schema and subsequent score interpretations across tool versions. The changes were designed to better detect true differences that inherently lie within classrooms that the first CLAC tool was potentially unable to measure.
Finally, the CLAC training process has been continually improved. Annual training is provided for observers where extra time is set aside for in-depth conversations on the operationalized definitions, scoring consistencies, and observing scenarios. Another revision employed is the randomization of the observers. Due to the logistics of collecting classroom observation data within a large-scale, multi-state intervention, on-site support staff often conducted the CLAC observations. While fully trained on the CLAC, we cannot know if the observers were unbiased and rule out a “halo effect” in their scoring.
Future research should examine the circumstances in which scores change across times of day, content focuses, and groupings and, relatedly, the generalizability of the scores. The CLAC observations were scheduled in advance with directions that any instructional activity was observable. Most classrooms (77%) included whole group instruction and only 3% of CLAC observations included routines. Connecting to other research, children often spend much less time in whole group instruction (23%) and substantially more time in routines (35%) (Early et al.
2010).
Moderator analyses should also be investigated. It is plausible, and even likely instructional groupings (whole group, free play, small group) affect student task-oriented learning. Higher levels of children’s engagement are associated with activity settings that allow a greater degree of choice, such as free choice (Vitiello et al.
2012). Similarly, certain content areas may more readily lend themselves to behaviors and instructional supports that foster engaged and active participation. The proportion of teacher-directed and child-initiated instruction may also moderate the relation between classroom task orientation and children’s learning. Finally, further review into the length of observation is recommended.
Moving forward, the CLAC has the potential to effectively guide and shape the classroom strategies and practices that promote student task orientation. While more evidence is needed to support the CLAC as a measure of classroom task orientation, evidence presented here suggests the CLAC connects to aspects of classroom quality, specifically the role of teachers in implementing effective practices.
Supporting task-oriented learning relies heavily on what classroom teachers believe, know, and ultimately do. Individual CLAC items that loaded onto the first factor, instructional responsiveness, often were observed measures of teachers’ direct and indirect teaching interactions and methods. Similarly, student engagement (factor 2) could be interpreted as the result of strategies teachers have employed that promote positive behavior management and classroom engagement. While evidence of the professional development (PD) interventions’ impact on task orientation is unknown, changes in specific teacher interactions have been observed across different learning domains, including language and literacy (McCollum et al.
2013; Piasta et al.
2012; Powell et al.
2010) and social-emotional interactions (Hamre et al.
2012; Hemmeter et al.,
2011; Raver et al.
2008). Further, the CLAC may be a valuable tool in providing data to inform a variety of classroom and programming interests. The CLAC may serve to inform broader program quality via progress monitoring or more summative evaluation. Additionally, the CLAC could potentially be used to assess the impact of specific interventions (e.g., those that target student engagement). Regardless of its application, it is imperative data from CLAC observations directly inform the very practices it measures.
In conclusion, findings show that the CLAC measures two dimensions of classroom context—instructional responsiveness and student engagement. Each dimension was independently associated with PreK learning gains. Findings enhance understanding of how effective classroom strategies and environments affect student learning through the development of self-regulation and task orientation. Facilitating students’ early self-control and self-directed behavior provides a strong foundation for learning (Fitzpatrick and Pagani
2013) that helps ensure that gains can be sustained as children transition to kindergarten and the elementary grades.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.