Measuring Acceptance of Block-Based Coding Environments

This is an instrumental study design (Ato et al., 2013). This design involves research that examines the psychometric properties of measurement instruments.

The sample was drawn using purposive sampling from 15 primary schools located in the province of Burgos (Spain). Participants attended a week-long intensive curriculum enrichment program, which included visual coding activities within science and math lessons. The BBC:MicroBit microcontroller board and its block-based environment MakeCode were used. Specifically, students programmed the BBC:MicroBit microcontroller boards into sensors that were used during hands-on laboratory investigations. For instance, a thermometer was programmed to measure the temperature of different water samples. Another example includes coding a timer to determine how long it takes to filter and purify the water (program detailed in Toma, 2020).

There were 337 elementary school students. After removing 22 questionnaires (6.5%) with missing data or more than one response per item, a valid sample of 315 students was formed. Almost half of the participants identified as girls (49.3%). Participants were enrolled in 3rd (n = 25), 4th (n = 77), 5th (n = 104), and 6th (n = 109) grades and had a mean age of 10.18 years (SD = 1.13).

A paper–pencil questionnaire was distributed on the last day of the program. To prevent social desirability, visiting school teachers were absent throughout data collection. Students were informed that their participation was voluntary and anonymous and that their responses would not affect school grades.

DeVellis (2017) guidelines for scale development were used. The first step was to determine the construct to be studied. As mentioned earlier, the TAM model was chosen. The second step was to generate a pool of items reflecting the latent variables of the TAM model. The recommended minimum number of items per construct is three. Therefore, six items measuring perceived usefulness, ease of use, and attitudes were developed. The items were worded to refer to the action of "coding" rather than to visual coding itself (Davis et al., 1989). The items on perceived usefulness related to different aspects (e.g., future study, school, life) to capture the phenomenon. The items on ease of use referred to students' perceived difficulty in using visual coding. Finally, the items measuring attitudes referred to positive dispositions toward visual coding. Therefore, simple terms (e.g., enjoyment, fun, or interest) were used. Some items were worded negatively to avoid acquiescence. In the third step, a five-point Likert scale (1: strongly disagree; 2: disagree; 3: Neither disagree or agree; 4: agree; and 5: strongly agree) was chosen as the measurement format. The fourth step was to assess the content validity of the items. Finally, in the fifth step, the revised items were administered to a large number of participants and tested for construct validity.

Inter-rater agreement in linking the items to the TAM constructs was high, with at least seven of the eight experts reaching consensus (87.5–100%). Inter-rater agreement on the ratings of each item for linguistic appropriateness and clarity was also adequate, with at least six of the eight experts reaching consensus (75–100%). Few changes, however, were proposed. The word 'irrelevant' was changed to 'of little use to me'. Likewise, 'relevant' was replaced with 'useful', which are easier words for Spanish-speaking elementary students.

The Kaiser–Meyer–Olkin measure of sampling adequacy was 0.858 and Bartlett's test for sphericity was statistically significant (χ² = 1442.397, p < 0.01), supporting the factorability of the data matrix. After parallel analysis, only three latent factors exceeded the eigenvalues of a randomly generated sample with the same characteristics (Fig. 3).

The EFA was then repeated, retaining three factors. Three items had commonalities below 0.30. After removing these items, two items with loading less than.40 and one item with cross-loadings emerged. Removal of these items yielded a final three-factor solution of conceptual meaningfulness that explained 58.46% of the variance (Table 1). The final questionnaire consisted of 12 items distributed as follows: four items measuring attitudes toward visual coding (34.52% variance explained); two positively and two negatively worded items measuring perceived usefulness of visual coding (14.89% variance explained); and three positively and one negatively worded items measuring perceived ease of use of visual coding (9.05% variance explained). The Spanish version of the items is included in "Appendix 1".

Items in each factor were moderately and highly correlated with adequate ranges for the attitudes (0.42 ≤ r ≤ 0.61); usefulness (0.32 ≤ r ≤ 0.59); and ease of use (0.30 ≤ r ≤ 0.48). Hence, factors exhibit convergent validity. Similarly, a moderate correlation between the attitudes and usefulness (r = 0.55), and a low correlation between attitude and ease of use (r = 0.27), and usefulness and ease of use (r = 0.22) provides evidence of discriminant validity of the retained factors.

Exploratory factor analysis by gender resulted in the same three-factor structure explaining 60.49% and 62.49% of the variance for the girls and boys sample, respectively (Table 2). However, one of the items measuring ease of use had a commonality below the 0.30 cutoff for the sample of boys. Since its removal affected reliability and given its acceptable factor loading, it was retained in the final questionnaire. Taken together, these findings provide evidence for construct validity.

Each factor exhibited adequate reliability. Cronbach’s alpha reliabilities were 0.82 for attitude, 0.75 for usefulness, and 0.76 for ease of use factors. McDonald’s omega coefficients were 0.83, 0.74, and 0.68, respectively. The Spearman-Brown split-half indices were 0.80, 0.67, and 0.73, respectively. Taken together, these findings provide evidence of good to excellent reliability for each retained factor.

To the best of the author’s knowledge, this study represents the first effort to advance an acceptance of block-based or visual coding instruments for the Spanish context. Except for Cheng, (2019), whose study includes Chinese students and uses ad-hoc measures, research on the acceptance of visual coding environments such as Scratch, Blockly, Snap, or Micro:bit Makecode is scarce. Hence, this investigation has taken an important step forward to bridging the gap in prior literature on visual coding.

Elementary school students are introduced to coding through block-based resources (Chalmers, 2018; Sáez-López et al., 2016; Sung et al., 2017). Based on the TAM model, students' behavioral intention to engage in coding-related activities depends on their perception of such resources as useful and easy to use. Ultimately, this will lead to positive attitudes and acceptance of visual coding (Al-Maroof et al., 2021; Davis, 1989). The proposed instrument is therefore likely to be useful in determining whether the visual coding environments that are widely used in elementary schools are, indeed, accepted by students. For example, studies comparing existing block-based resources may be informative as to what coding resources are preferred by students of different ages and gender (Yildiz Durak, 2020). Furthermore, research into the features of such resources that can improve the perceived ease of use, usefulness, and students' attitudes may also be beneficial to the computational thinking research agenda. This could ultimately lead to the refinement and improvement of visual coding environments.

It is also beneficial to investigate the psychometric properties of the instrument in secondary school students. This would contribute to the development of cross-sectional and longitudinal studies that would provide a broader picture of students' acceptance of visual coding resources.

The present investigation does exhibit some limitations. First, it should be noted that the psychometric properties of the instruments were tested with students engaging with one specific visual-coding resource (i.e., Micro:bit MakeCode). Therefore, the latent structure of the instrument should be confirmed in future studies using alternative visual coding environments (e.g., Scratch).

Second, a school-year difference in sample size was also noted. Children from the last four years of Spanish primary school (3rd to 6th grade) were included. Yet, third-graders made up fewer than 8% of participants. This has hampered additional analysis to establish if the factor structure is invariant across school grades. Future studies with a larger sample size are warranted (Gaskin & Happell, 2014).