Psychometric Validation of the Robotics Interest Questionnaire (RIQ) Scale with Italian Teachers

Self-efficacy, while a difficult construct to assess adequately (Tschannen-Moran & Hoy, 2001), has also started to receive international attention from an educational perspective. Bandura (1994) defined it as the strength of a subject’s belief and confidence in his or her ability to organise and complete a task through self-measurement of the degree to which he or she evaluates task performance. This system influences personal perceptions of situations and consequently the behaviours enacted by the individual in response to various contextual inputs. Bandura (1977) distinguished between outcome expectancy, or the estimation that an action will lead to a certain outcome, and efficacy expectancy, or the very belief that one is capable of performing an action to achieve a predetermined goal. In line with Bandura’s social cognitive theory, Tschannen-Moran et al. (1998) proposed an integrated model in which teachers’ perceptions and competence positively influence competence itself, which can affect students’ academic success, from the quality of teaching to the instructional strategies implemented.

A high level of teacher self-efficacy has also been associated with positive perceptions of collaboration, openness to innovation, and readiness for continuing education (Collie et al., 2012; Mannila et al., 2018; Ninković & Knežević Florić, 2018; Pas et al., 2012). Teachers’ self-efficacy is critical for the development of digital and computational skills to provide students with current and up-to-date instruction (Mannila et al., 2018). There is a need to support teachers in developing self-efficacy in digital skills: studies have shown that good levels of self-efficacy in the discipline of teaching support lifelong learning improve the learning environment and discourage states/syndromes of burnout. Bandura (1994) noted that the degree of self-efficacy changes throughout life, and it can be influenced in multiple ways, such as by experiences of competence, observation of competent social role models, social persuasion that one can be competent, and verbal feedback. Considering that most teachers called to in-service training have not had enough practice in teaching ER, observation of models and social feedback seem to be factors that can improve the self-efficacy beliefs not only of in-service teachers but also of teachers still in training.

According to some scientific evidence (Tschannen-Moran & Hoy, 2007), contextual factors (e.g. peer support and teaching resources) promote the construction of teachers’ self-efficacy and are particularly influential in the early years of service. For this reason, instruments measuring teacher self-efficacy are relevant if administered as early as pre-service and university training, collaterally to internship and/or laboratory activities. In recent years, numerous research studies have shown how teachers’ perceptions and their sense of self-efficacy can in fact influence learning processes. Improving one’s ability to implement teaching, learn new strategies, and better present teaching content has, therefore, a significant impact on motivation, school inclusion, and, consequently, on learning itself (Giannandrea et al., 2021). Jaipal-Jamani and Angeli (2017), in line with the above, asserted that engaging with robotics as part of a training course can improve teachers’ perceptions of self-efficacy and encourage the implementation of ER activities to teach science disciplines and computational thinking (CT) skills.

Summaries of the literature published in the last decade on ER (e.g. Benitti, 2012; Bonaiuti et al., 2022a, b; Kyriazopoulos et al., 2021) have demonstrated the increase in research data, making explicit the educational purposes, areas of application, and scientific evidence in terms of the educational effectiveness of the practice itself. Robots in the educational context have emerged as a viable tool since the time of Papert, who called it an ‘object with which to reason’ (Harel & Papert, 1991), borrowing Winnicott’s (1971) concept of the transitional object. The robotics artefact still proves valuable today as a mental tool (Mikropoulos & Bellou, 2013) for the acquisition of twenty-first-century (Alimisis, 2013) and STEM skills (Benitti, 2012; Ching et al., 2019; Nugent et al., 2009; Sullivan, 2008; Toh et al., 2016), as robots can boost motivation and improve student learning and achievement (Anwar et al., 2019; Athanasiou et al., 2019; Lancheros-Cuesta & Fabregat, 2022; Scaradozzi et al., 2015; Vlasopoulou et al., 2021).

The use of robots in education has also supported the inclusion of students with special educational needs and the protection of gender equality (Daniela & Lytras, 2019; Sullivan & Bers, 2019; Syriopoulou-Delli & Gkiolnta, 2021). Indeed, several researchers have argued that interacting with robots also develops social skills (Mitnik et al., 2008; Owens et al., 2008; Toh et al., 2016), teamwork skills (Johnson, 2003) and cooperative learning skills (Nourbakhsh et al., 2005). It follows that fostering ER in schools and training teachers to design and implement such activities, as the directions and school curricula of several nations require, is a priority for the new millennium. ER activities, compared to traditional teaching, actively involve pupils, making teaching more engaging (Eguchi, 2014). In their review ‘teacher training on educational robotics’, Giannandrea et al. (2021) stated that pre-service and in-service training allowed teachers not only to readjust and integrate knowledge about robotics and teaching methodologies (essential for acquiring specific skills) but also to actually understand the benefits of technological apparatuses in the pedagogical context. In fact, it turns out that there are numerous studies that have demonstrated the importance of interest and knowledge about content to teach it well (Hattie, 2012; Shulman, 1987). Teachers’ attitudes, perceptions, and beliefs about ER are based on such understandings, which can shape as stimuli or act as a limitation to the implementation of the practice itself (Hew & Brush, 2007; Lawson & Comber, 1999). A study by Papadakis and Kalogiannakis (2021) has shown that a teacher’s positive attitude about robotics practice is positively associated with ER knowledge.

There are numerous works in the literature that emphasise the importance of collaboration between teachers. In a systematic review of the relationship between teacher collaboration and student academic success, García-Martínez et al. (2021) revealed that the most widely used collaborative modalities were related to instructional processes and improving student academic performance. They pointed out that although building a collaborative climate is demanding and challenging, collaborative support between teachers is crucial to school success. Vangrieken et al. (2015), in an earlier systematic review, showed that the observed benefits of teacher collaboration in the literature can be found at three different levels: student, teacher, and school. The main advantages reported at the teacher level include teachers experiencing the following: being more motivated, a decreased workload, a positive impact on teacher morale, greater efficiency, increased communication, improved technological skills, shared instruction strategies, and teaching suggestions and reduced personal isolation. At the student level, it was reported to improve understanding and performance. At the organisational level, communication, openness, and participation are key for creating a climate of trust so that the benefits reported include a positive influence on the perception that the school climate is supportive of innovation, better adaptation, and more innovation, a cultural shift to more equity, a school-wide attention to the needs of students, a flattened power structure, and the fostering of a professional culture of intellectual enquiry. In other words, teacher teamwork is a force that positively influences the whole school community (Mora-Ruano et al., 2019). The TALIS 2013 survey found that collaboration between teachers is among the factors that can boost teacher job satisfaction (OECD, 2014), which is a core element of an effective teacher (Mora-Ruano et al., 2019).

Professional relationships based on trust contribute to the development of a common vision for the school (García-Martínez et al., 2021). Teachers who work collaboratively also increase their effectiveness and expertise (Hattie, 2015), helping to improve self-efficacy (Puchner & Taylor, 2006). Bandura himself (1977) recognised the fact that individuals do not work as social isolates and that people form beliefs about the collective capabilities of the groups to which they belong. He defined collective efficacy as ‘a group’s shared belief in its conjoint capabilities to organise and execute the courses of action required to produce given levels of attainments’ (p. 477). As Klassen et al. (2011) stated, teachers’ collective efficacy refers to the beliefs that teachers possess in their collective the capabilities to help students develop and learn. It is reasonable to assume that beliefs about the ability and effectiveness of the teacher group in a school to cope with a variety of challenges also support and nurture beliefs about personal effectiveness. It is no coincidence that Klassen et al. (2011), in a study that explored the relationship between teachers’ collective efficacy, job stress, and job satisfaction, stated that when teachers experience challenges and setbacks that may lower their individual motivation, these setbacks may be mitigated by beliefs about their colleagues’ collective ability to affect change. Given the close link between teamwork and self-efficacy and between the latter and student learning outcomes, one dimension analysed by the questionnaire is precisely related to collaboration between teachers. The positive effects of teamwork by teachers engaged in STEM teaching are also strongly confirmed by a specific synthesis of the literature.

Problem-solving is a decisive factor for success in teaching. Problem-solving promotes meaningful learning, knowledge transfer, metacognitive skills, engagement, critical thinking, and collaboration among students. It helps students connect existing knowledge to real-world situations, develop problem-solving abilities, and enhance their overall learning experience when supported by effective teaching strategies. Through meta-analyses of the literature, Hattie (2012) found that the ability to solve problem in teaching has a high effect size (d = 0.61) and is capable of significantly improving students’ learning. Problem-solving leads students to reason about the process leading to the solution, the validity of the solution, and the possible alternatives, as well as enabling them to reflect metacognitively on their own strategies. Problem-solving, by challenging students to solve real problems or authentic learning situations, makes learning motivating and, because it often requires group work, promotes peer learning and constructive discussion.

Papert (1980), among the first to identify the potential of robotics to develop children’s thinking skills, identified the promotion of problem-solving as one of the strengths of this type of experience. From this perspective, having a robot (virtual or real) explore the world around it requires the ability to plan a series of actions by predicting their consequences in view of a given goal to be achieved and allows students to formulate and then to select, from among the many ideas and models available to them, the most functional ones. Giving students the opportunity to programme the behaviour of virtual agents in an unknown context is also a way to develop CT. This concept was taken up by Wing (2006) and later by others (Brennan & Resnick, 2012; Chalmers & Nason, 2017; Kazakoff Myers & Bers, 2014) in terms of its implications for students’ ability to engage in abstraction, decomposition, verification, debugging, and modelling during learning processes involving building, programming, and interacting with robots. In addition to the National Research Council (2010)—and after Wing—many other authors (Brennan & Resnick, 2012; Catlin & Woollard, 2014; Piedade et al., 2020) have taken an interest in this topic.

CT has emerged as a multifaceted construct, defined as a set of skills that enable us to think like a computer scientist when faced with a problem: thus, it involves posing, finding, and solving problems; abstraction and logical thinking; pattern recognition; and the ability to effectively manage information across technologies. As reported by Selby and Woollard (2013), Hoppe and Werneburg (2019), and the International Society for Technology in Education (ISTE) and the Computer Science Teacher Association (CSTA), the operational definition of CT in K–12 is a problem-solving process that includes (but is not limited to) the ability to think in abstractions, in terms of decomposition and algorithmically, and in terms of evaluations and generalisations. These skills are supported by attitudes that take the form of essential dimensions of CT: for example, confidence in dealing with complexity, persistence in working with difficult problems, the ability to deal with open-ended problems, and the ability to communicate and work with others to achieve a common goal or solution. CT therefore represents a type of analytical thinking that has many similarities with mathematics (problem-solving), engineering (design), and science (systematic analysis) but at the same time is useful in supporting and organising reasoning in numerous other disciplines, to the point of suggesting its introduction in the early childhood curriculum (Bers, 2021). However, CT does not turn out to be a discipline per se in K–12 education, but rather an aptitude—that is, a universally applicable set of skills that are essential for everyone in the twenty-first century, while being a form of cross-curricular knowledge that inhabits computing contexts. ER activities have the potential to foster CT’s unique skills because robotics requires students to interface with and explore algorithms, modules, sequences, loops, and variables (Sullivan et al., 2017). Bers (2021) expanded the concept of CT by considering it no longer just a problem-solving process, but rather an expressive process that can incentivise new modes of communication. There are numerous examples that require computational skills but are not related to the act of programming; Wing (2008) provided some related to everyone’s daily life—think of sorting and classifying Lego pieces or the procedure of executing a recipe. The enhancement of such skills is thus necessary to help students consciously experience daily life and to form future citizens of a technology-driven society.

A total of 823 teachers at public Italian schools were recruited (age mean 48.63 years, SD = 9.22; 19.2% males; mean of years of seniority = 17.67 years, SD = 11.41). They worked in the following scholastic levels (indicating the grades for the different levels): kindergarten, age 3–5 years (n = 90, 10.9%); primary school, grade 1–5, age 6–10 years (n = 291, 35.4%); secondary school of the first order, grade 6–8, age 11–13 years (n = 185, 22.5%); and secondary school of the second order, grade 9–12, age 14–18 years (n = 257, 31.2%). Among the participants, 277 teachers (33.7%) had access to their teaching position with a diploma (based on Italian legislation prior to 2011), and 546 teachers (66.7%) started their professional careers on the basis of a bachelor’s degree.

In relation to work areas, the teachers stated that their training was characterised as follows: teaching diploma (34.1%), primary education (7.9%), training in the humanities (20.2%), training in an expressive field (3.8%), and STEM (34.0%). Among these teachers, 22.7% declared having a further teaching specialisation. The participants worked in different Italian geographic areas: north (n = 229, 27.8%), centre and the island of Sardinia (n = 233, 28.3%), and south and the island of Sicily (n = 361, 43.9%). In the total sample, 68.8% of participants declared they did not have any experience in robotics.

We divided the participants randomly into two sub-samples to carry out two studies: specifically, for study 1 (EFA), sample 1 was composed of 412 participants (age mean = 48.79 years, SD = 9.15; 17.5% of males; mean of years of seniority = 18.08, SD = 11.33); for study 2 (CFA), sample 2 consisted of 411 participants (age mean = 48.47 years, SD = 9.29; 20.9% males; mean of years of seniority 17.25, SD = 11.49). Among the participants in study 1, 147 teachers (35.7%) had accessed their teaching position with a diploma (ante 2001), and 265 (64.3%) started their professional career on the basis of a bachelor’s degree. Regarding study 2, 130 teachers (31.6%) accessed their teaching position with a diploma (ante 2001), and 281 (68.4%) started their professional career on the basis of a bachelor’s degree. The detailed descriptive statistics for all participants, for sample 1 and sample 2, are presented in Table 1.

Non-probabilistic sampling was carried out; participants were recruited on a voluntary basis. Purposive sampling (Wolf et al., 2016) was applied, starting from the national list of schools furnished by the Italian Ministry of Education. Specifically, the survey was disseminated via a call to the managing head teachers at schools of all levels, randomly extracted from different geographical areas in Italy (north, central, south). The managers were informed of the scientific objectives of the survey, and after they had agreed to participate, all of their teaching staff received a link to the survey (administered by the digital platform LimeSurvey).

The participants were informed of the study’s objectives and were assured that their responses would remain confidential (informed consent was obtained from all participants). In conducting the present research, all procedures performed were in accordance with the guidelines and ethical standards of the institutional research committee.

The RIQ (Dorotea et al., 2021) evaluates four dimensions through 27 items measured on a five-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). The first dimension assesses teachers’ interest in robotics and technologies (items 1–3; an example item is ‘I find it interesting to learn about robots or robotics technology’); the second dimension is problem-solving practices (questions 4–11; an example of an item is ‘I try new methods to solve a problem when one does not work’); the third is working collaboratively (assessed by items 12–15; an example item is ‘I like to work with others to complete projects’); and the fourth dimension is self-confidence and knowledge to use robotics in classroom activities (items 16–27; an example item is ‘I have sufficient knowledge about robotics for use in teaching and learning activities’). In their original Portuguese version (Dorotea et al., 2021), the Cronbach’s alpha reliability was 0.950.

To outline the Italian version of the RIQ, we applied a back translation procedure, aiming to safeguard the cross-cultural equivalence in the translation of the items in different countries and cultures (Brislin, 1970). The RIQ in its Italian version can be found in the Appendix.

The teacher’s self-efficacy was measured using the QAI (La Marca & Di Martino, 2021), an Italian validated instrument with 25 items. Participants’ views about these dimensions were evaluated on a 6-point Likert scale, from 1 (strongly disagree) to 6 (strongly agree). The scale reliability was 0.959.

Analyses were conducted by applying a multiple-step method. With sample 1, we carried out study 1, through the application of EFA. To solve the problem related to identifying the number of factors to retain in the EFA, we applied the scree plot method (Cattell, 1966) and parallel analysis (Horn, 1965). To detect the factor structure underlying the Italian version of the RIQ, EFA was applied using principal axis factoring, with the Oblimin rotation. We measured the appropriateness of these data using the Kaiser–Meyer–Olkin measure (KMO) and Bartlett’s test of sphericity. We computed Cronbach’s alpha to assess the reliability of factors (Chiorri, 2023).

In study 2, CFA was applied. The factorial structure highlighted by the EFA and hypothesised by the authors of the RIQ (Dorotea et al., 2021) (four correlated factors) was evaluated in sample 2. Additionally, Pearson’s r correlation was computed between the scales of the RIQ instrument and the QAI (La Marca & Di Martino, 2021), to evaluate the convergent validity of the RIQ.

It might be appropriate to recall here that, in psychometrics, validity corresponds to the capability of an instrument to indeed measure what it intends to measure. It is not a unique concept, but a multidimensional one; in fact, it is possible to distinguish different types of validity which correspond to different methods of verification (Chiorri, 2023). Construct validity is given by the relationship of the actual measurement to the abstract dimension it is intended to measure (in our case, teachers’ perceptions of ER). Among the several aspects considered by construct validity, there is convergent validity, which in our case was assessed by calculating the correlation between the factors of the RIQ and different measures (accomplished with an already validated instrument) of a related construct (specifically, teachers’ self-efficacy, assessed by the QAI). To prove convergent validity, we assumed we would find a significant correlation between the RIQ factors and the QAI scale. This evaluation was applied by Pearson’s r correlation coefficient, which is a parametric inferential bivariate statistical technique useful to quantify the relationship between two continuous variables in terms of magnitude and direction. This coefficient is standardised, ranging from − 1 to + 1, and is associated with a level of statistical significance, which, when it is lower than the critical alpha value (p < 0.05), represents the presence of a statistically significant correlation, or association, between the two variables (Chiorri, 2023).

Furthermore, to establish criterion-related validity, the RIQ scores were compared across different groups of teachers who participated in this investigation. These evaluations were made by applying multivariate analysis of variance (MANOVA), which is a parametric multivariate inferential statistical technique that compares the means of different groups of participants (Tabachnick & Fidell, 2007). MANOVA evaluates the differences between the means of groups identified on the basis of one or more factors, taking into account several continuous dependent variables; the latter are grouped in a weighted linear combination (in our case, the four factors of the RIQ). MANOVA evaluates whether the newly created combination differs in different groups identified by the factor (in our case, expertise in ER), essentially investigating whether the combined variable shows any systematic differences between the means of the groups.

The MANOVA results are associated with a level of statistical significance, which, when it is lower than the critical alpha value (p < 0.05), denotes the presence of statistically significant differences between groups. In MANOVA, potential differences between the averages of the four RIQ factors were evaluated in relation to the between-subjects factor defined as ‘expertise in robotics’ (experienced/inexperienced teachers; Tabachnick & Fidell, 2007).

The analyses were applied using the open-source software Jamovi (release 2.2.5) and Jasp (release 0.16.3).

In study 1, we applied the EFA (principal axis factoring) with Oblimin rotation (KMO measure of sampling adequacy = 0.943; Bartlett’s test of sphericity chi-square = 9647; df = 351; p < 0.001). The four components explained 65.2% of the total variance, and all items had a component load of 0.40 or above (Table 2). The internal consistency was determined by Cronbach’s alpha, and it showed good values (factor 1, 0.971; factor 2, 0.887; factor 3, 0.835; factor 4, 0.818). In Table 2, we can observe that the factorial structure of the Italian version of the RIQ is consistent with the four correlated factors identified by the authors of the instrument in their original Portuguese version (Dorotea et al., 2021).

In the second sample, CFA (estimator diagonally weighted least squares, DWLS, robust method) (Rhemtulla et al., 2012) was applied; we examined the data fit in relation to the a priori model (four correlated factors) specified by the authors of the RIQ and highlighted in the previous EFA. To assess the model, multiple indices were considered: ratio of chi-square and its degrees of freedom defined as being good if it is below three (Schermelleh-Engel et al., 2003); comparative fit index (CFI) for which higher than 0.90 is considered suitable (Byrne, 2001); and the indices root mean square error of approximation (RMSEA) and standardised root mean square residual (SRMR), for which values lower than 0.08 are designated as satisfactory fit (Hu & Bentler, 1999). The CFAs applied showed good fit indices (see Table 3) and significant parameter estimations, confirming the four-factor structure of the questionnaire in this Italian version (see Table 4).

The scales of the RIQ instrument were correlated with a measure of teacher self-efficacy (QAI; La Marca & Di Martino, 2021) to appraise the convergent validity of the RIQ. The linear correlations (Pearson’s r) between the assessed variables highlighted and confirmed the relationships between the dimensions examined, consistent with the literature (Table 5). Indeed, as assumed on the basis of previous studies on this topic, the three factors of the QAI (evaluating self-efficacy in teachers) showed systematic, direct, and statistically significant linear correlations (p < 0.05) with each factor evaluated by the RIQ (specifically, the closer the value of the correlation coefficient approached to one in absolute value, the closer the relationship between the two variables considered). This means that as the scores obtained in the QAI increased, the scores obtained in the measurements carried out with the four factors of the RIQ also increased proportionally (or else, as the scores obtained in the QAI decreased, the scores obtained in the measurements carried out with the four factors of the RIQ also decreased proportionally).

Additionally, to estimate criterion-related validity, the scores of the RIQ factors were compared across different groups of teachers who participated in this investigation; we assessed the differences/similarities in the mean scores of the RIQ by applying MANOVA (Table 6). In this multivariate statistical analysis, we considered as a between-subjects factor the variable expertise in robotics (experienced/inexperienced teachers), identified by the self-evaluation in two items of the questionnaire.

The findings highlighted a significant multivariate effect of expertise in robotics (Wilk’s lambda = 0.550; F = 167.239; df = 4, 818; p < 0.001; partial eta squared = 0.450); the significant effect was confirmed at the univariate level (see Table 6). More specifically, we observed a significant effect of expertise in robotics for all four dimensions of the RIQ, in which teachers with experience in robotics showed scores significantly higher than their colleagues without experience (RIQ factor 1: F = 646,385; df = 1, 821; p < 0.001; partial eta squared = 0.441; RIQ factor 2: F = 139.626; df = 1, 821; p < 0.001; partial eta squared = 0.145; RIQ factor 3: F = 19.969; df = 1, 821; p < 0.001; partial eta squared = 0.024; RIQ factor 4: F = 52.315; df = 1, 821; p < 0.001; partial eta squared = 0.060) (Table 6). These data emphasised that the teachers showed significant differences in their scores in the RIQ based on their experience in ER. These differences between means are graphically represented in Fig. 2, where we can observe that the experienced teacher in ER showed statistically significantly higher scores for factors of the RIQ (F1, Self-confidence and knowledge; F2, STEM and ER interest; F3, Teamwork; and F4, Problem-solving) than their inexperienced colleagues (see Table 6 and Fig. 2).