A research of analysing the effectiveness of speaking-pen on English learning in consideration of individual differences using a linear mixed-effect model

In an era of globalization, the Ministry of Education, Culture, Sports, Science and Technology in Japan (MEXT 2011) began implementing ‘Foreign Language Activities’ as a compulsory class beginning in fifth grade, which is based on the new courses of study that were introduced in April 2011. The ‘Foreign Language Activities’ aim to familiarize children with English by focusing on intonation and pronunciation, listening and speaking. The MEXT (2014) has considered beginning ‘Foreign Language Activities’ in third grade which aims to improve ‘listening’ and ‘speaking’ skills, and implementing an ‘English Course’ in fifth grade, which aims to not only improve ‘listening’ and ‘speaking’ skills, but also ‘reading’ and ‘writing’ skills.

They have also proposed adopting effective ICT materials in order to help children recognize alphabetical letters and notice differences in intonation, characteristics, and structure between Japanese and English as a guide for the teaching support material. Many researchers study and survey teaching support materials used for teaching elementary school children.

Although many studies on tools used for early English education have been conducted, there are few studies on English education that have analysed longitudinal learning data of children using tools and modelled the effectiveness of the tools in consideration of individual differences.

According to Pourhosein Gilakjani (2017), technology assists learners in adjusting their own learning process and they can have access to a lot of information that their teachers are not able to provide. Parvin and Salam (2015) carried out a study and declared that by using technology, learners get the chance to increase their exposure to language in a meaningful context and make their own knowledge. Pourhosein Gilakjani (2014) maintained that using technology can create a learning atmosphere centered around the learner rather than the teacher that in turn creates positive changes.

However, their papers did not analyse longitudinal learning data of children using tools and model the effectiveness of the tools in consideration of individual differences.

On the other hand, there are many educational studies that have used the Linear Mixed-Effect Model (LME) in order to take individual differences into account in the modelling of the educational effectiveness. In Japan, Kawaguchi (2009) propose to use a LME to analyse school effects. His models include school level variables and children level variables as fixed effect and random effect variables accompanying fixed effects. In other countries, Xu, Yuan, Xu, and Xu (2014) analysed Chinese high school students’ time management with regard to their math homework using the LME. Their models depict class level and children level variables such as ‘Motivation’, ‘Arranging environment’, ‘Family homework help’, and ‘Gender’, and they analysed these factors in detail. Hsu and Kuan (2013) explore the factors that influence the elementary or junior high school teacher ICT integration by analysing a detailed model according to the level of schools and teachers in Taiwan. Roman and Murillo (2012) used the model to analyse achievements in math and language of the elementary school in Latin America according to factors such as country level, school level and family socio-economic level. Kwok, Lai, Tong, Lara-Alecio, Irby, Yoon and Yeh (2018) analysed complex longitudinal data of project of English Language and Literacy Acquisition (ELLA) in educational research.

Although many educational studies using the LME have been conducted, there are few studies that have analysed longitudinal data according to detailed modelling of individual differences of children, for students’ detailed English educational experience.

Therefore, the purpose of this study is to analyse longitudinal data in order to propose modelling the effectiveness of a Speaking-pen in support of four English skills (reading, writing, listening, and speaking) in consideration of individual differences depending on the detailed experiences of English learning. The learning that was conducted during the investigation utilized two tools; namely, a speaking-pen and an audio CD, which were used to enhance English learning. Using a speaking-pen, children can learn English in a similar way as one uses a pencil, without prior knowledge of, and preparation for, PC. This method differs from learning using CALL materials, which are generally used for English education. The speaking-pen adopted in this study has an extraordinary function. It can record and play back users’ voices in addition to its conventional function, in which English pronunciations are already recorded. Users can compare between their own voices and English pronunciations already recorded (the speaking-pen was made by Gridmark Inc.).

In section 2, the investigation method of this study is explained. In subsections 2.1, 2.2, and 2.3, we describe the experimental design, the construction of the test, and the construction of textbook, respectively. In section 3, we propose modelling the effectiveness of a speaking-pen in support of four English skills in consideration of individual differences depending on the detailed experiences of English learning. In section 3.1, the model of the total score is shown, and in section 3.2, the model of each of four skills is shown. In section 4, the results of the effectiveness of speaking-pen based on each model of total score and four skills are shown and discussions are presented.

The research for this study was conducted from October 2013 to March 2014. Ninety second-grade private school students at Shukutoku elementary school participated in this study with the consent of their guardians. In this school, children learn English twice a week beginning in first grade. They learn it by focusing on conversation skills with a native English teacher. Therefore, they have high proficiency in spoken English, which increases their motivation to continue learning. This differs from children in general public elementary schools. In this study, a two-period (2 × 2) cross-over design was adopted as the experimental design. The children were divided into two groups in which both groups were able to use both a speaking-pen and an audio CD in different periods. A pre-questionnaire was implemented before the research experiment began. The pre-questionnaire included some items for investigating the children’s English learning background, which can be found in Chapter 3. Based on the pre-questionnaire, the children’s responses were categorized into eight categories in accordance with the results of three categories of responses, ‘Experience of English Learning’, ‘Practice of Home Learning’ and ‘Experience of Using a Speaking-Pen’. The children in each group were allocated to two groups using a Bernoulli trial, in which the probability of success was set at 0.5 as the Bernoulli probability parameter, so that there would be no differences among the children’s background between the two groups. As a result, 45 children were assigned to Group 1 and the other 45 children were assigned to Group 2 (Fig. 1).

Bild vergrößern

The children in Group 1 learned in their home using a speaking-pen during the first six weeks of the investigation (the first period), and using an audio CD during the last six weeks (the second period). Both six-week sessions were separated by a four-week long inactive term. The children in Group 2 learned in their home using an audio CD during the first period and using a speaking-pen during the second period. In terms of home learning during the investigation, the children were not forced to use either a speaking-pen or an audio CD. Their learning conditions, which included frequency, time, and the means of use depended on their own independence of will and volition to learn. There are four achievement tests that measured the initial skills or improvements in their learning. The first test was implemented before the first period, the second test was implemented after the first period, the third test was implemented before the second period, and the fourth test was implemented after the second period. A post-questionnaire was administered to the children after the research experiment was complete. The post-questionnaire included some items for investigating timing, frequency and the hours of use. This can be seen in Chapter 3.

Bild vergrößern

The construction of four tests looked similar to each other. This paper cites the second test in the explanation. All of the tests are composed of six sections, and their total scores add up to 100. In terms of the first test, we referred to the previous study conducted by Tsubaki, Gonda, Kato and Maeda (2015). In Part 1 of the test, after the children read the spelling of a word and see an accompanying picture, they connect the word and the pictures with a line. This is considered to be an appropriate test for measuring a child’s reading ability. This section has fifteen questions and one point is given per one accurate combination so that fifteen possible points can be earned in total. In Part 2 of the test, after the children see a picture, they fill in the blank with one letter for each question. This is considered an appropriate test for measuring a child’s writing ability. This section has ten questions, and two points are given for each correct answer. In Part 3 of the test, after the children read a question and see a picture, they choose the correct answer. This is considered to be an appropriate test for measuring a child’s reading ability. This section has five questions, and two points are given to each correct answer. In Part 4 of the test, after the children hear a question, they choose an appropriate answer sentence. This is considered an appropriate measurement of a child’s listening ability. This section has five questions, and four points are given to each correct answer. In Part 5 of the test, after the children listen to a sentence that contains one blank in the place of a missing word, they fill in the blank with a letter. This is considered an appropriate measurement of a child’s listening and writing abilities. This section has five questions, and four points are given to each correct answer. In Part 6 of the test, a native English teacher asks each child three questions in English, and each child answers the question in English. This is considered an appropriate measurement of a child’s speaking ability. Some examples of questions are, ‘Is this a dog?’ (accompanied by a picture of a dog); ‘What colour is this?’ (accompanied by a picture of a yellow cat); ‘What’s this?’ (accompanied by a picture of an umbrella). (Fig. 3).

Bild vergrößern

The textbook is composed of four units, and each unit is composed of seven sections. In the first period, the children study from the first and second units of the textbook, and in the second period, the children study from the third and fourth units. In this section, we refer to the second unit in order to describe the components of the textbook. One may refer to the study by Tsubaki et al. (2015) for a further understanding of the first unit. The components of the four units in the textbook are similar.

In Section 1, the children learn basic conversational phrases that align with the theme of the unit. The children read and listen to conversational questions and their corresponding answers, such as ‘What colour is this?’; followed by the response: ‘It’s blue.’ They can compare their pronunciation with the native English speaker’s when they use the speaking-pen to record their pronunciation.

In Section 2, the children learn a set of words that corresponds with the theme of this unit. The theme of the second unit is colour. The children learn pronunciations of colour words, such as ‘red’ and ‘yellow’. The themes of the other units are animals, food, and a birthday party. In this section, children can practice their pronunciation as they did in Section 1.

In Section 3, the children can listen to question sentences that correspond with the theme of the unit and choose correct answers after seeing a set of pictures. In this unit, the children can learn the words of colours by listening to their names. In the other units, the children can learn their numbers, as well as how to answer ‘yes’ or ‘no’.

In Section 4, after the children listen to a word, they use the speaking-pen to spell the word. This section is only available to the children who use the speaking-pen. For example, the children in Group 1 can practice spelling words during the first period, and the children in Group 2 can practice spelling words during the second period.

In Section 5, after the children see pictures of objects and listen to their corresponding names, they can practice writing the correct spelling of the words.

In Section 6, after the children read question sentences and listen to questions using a speaking-pen, they can practice choosing correct answers. This section contains two questions and is only available to the children who use the speaking-pen.

In Section 7, after the children listen to a group of words that align with the theme of this section, they can practice spelling the words. In terms of using the speaking-pen, they can learn the correct pronunciations of words by comparing their own pronunciations with those of a native English speaker.

The Common European Framework of Reference for Languages (CEFR) is a set of guidelines used to describe the achievements of students of foreign languages throughout Europe. In Japan, CEFR-J is based on CEFR, has been proposed by Touno et al. (2010, 2012a, b), and was adjusted for Japanese English learners. CEFR-J descriptions corresponding to each unit of the test and text in this study are shown in Table 1. In the first column, reading, writing, and listening are denoted as R, W, and L, respectively. Speaking is divided into two ability categories. S1 refers to ‘Spoken relationship’, and S2 refers to ‘Spoken production’. In the second column, proficiency levels are arranged in numerical order. For example, if we consider listening, PreA1 corresponds to ‘perception of pronunciation with which Japanese are familiar as a katakana word (loan word)’; A1.1 corresponds to ‘greeting, name, date, day of the week, numbers, words, and expressions which are used in daily life’; A1.2 corresponds to ‘words, short sentences, question, familiar and personal requests and preferences (like or dislike, route guiding, etc.)’; and A1.3 corresponds to ‘informal speech in daily conversation (personal questions, daily instructions, requests, etc.)’. In the third and fourth column, the numbers correspond to the section number in the test or textbook respectively.

To illustrate this table, the sections of the test and text principally focus on the A1 level of ‘a beginner who just began learning English’.

In this section, we propose a Model (T) of the total score.

The total score is modelled as follows:

y_ij(k) is the total test score of the child i at time j with tool k. δ_i is defined as the parameter of each child i, β_j as the effect of time, γ_k as the effect of the tool, and βγ_jk as the interaction between time j and tool k. ε_ij(k) is the error.

Furthermore, the parameter δ_i of each child i in Eq. (1) is modelled by the variables whose clear trends were observed significantly using a one-way ANOVA of the total score of the first test.

The parameter μ is defined as ‘Mean over individual’, α₁ as fixed effect of ‘Gender’, α₂ as fixed effect of ‘Private English School’, α₃ as fixed effect of ‘Tutor’, α₆ as fixed effect of ‘Speaking-pen Experiences’, α₇ as fixed effect of ‘Home Learning’, α₈ as fixed effect of ‘Homework from Private English School’ and α₉ as fixed effect of ‘Favour’. The parameter ω_i expresses ‘Individual Differences’ and is assumed to be \( {\omega}_i\sim N\left(0,{\sigma}_{\delta}^2\right) \).

The effect β_j of time j in Eq. (1) is modelled by the variables whose clear trends were observed significantly using a one-way ANOVA of the total improvements in test scores.

The parameter π_j is defined as ‘Mean of Time j’ and α_1j as ‘Gender × Time j’, which means the fixed effect of ‘Gender effect at Time j’.

The effect γ_k of tool k in Eq. (1) is modelled by the variables whose clear trend was observed significantly using a one-way ANOVA of the total test score improvement.

The parameter ο_k is defined as ‘Mean of Tool k’ and α_1k as ‘Gender × Tool k’, which means the fixed effect of ‘Gender effect using Tool k’.

The effect βγ_jk in Eq. (1) is modelled by the variables whose clear trends were observed significantly using a one-way ANOVA of the total improvements in test scores.

The result of the one-way ANOVA showed that ‘Frequency’ was significant regarding the improvement of first period students, and ‘Gender’ was significant regarding the improvement of second period students. We considered this to be the result of interactions between these variables and the effect of time. Accordingly, these interactions are included in the model.

The parameterξ_jk is defined as ‘Mean of Time j × Tool k’, α_1jk as ‘Gender × Time j × Tool k’, which means the fixed effect of ‘Gender effect in Time j × Tool k’, and α_10jk as ‘Frequency × Time j × Tool k’, which means the fixed effect of ‘Frequency effect in Time j × Tool k’.

In this section, we propose the models of four English skills. The four models are constructed in the same way as Model (T).

Reading score is modelled using parameters in Table 2 as follows:

Writing score is modelled using parameters in Table 2 as follows:

Listening score is modelled using parameters in Table 2 as follows:

Speaking score is modelled using parameters in Table 2 as follows:

Model (R), Model (W), Model (L), and Model (S) are adopted among all the possible models regarding all combinations of variables chosen using a one-way ANOVA. Comparing all the possible models’ value of AIC, the models that had the minimum value of AIC were adopted.

The variables and parameters composing all models indicated in Section 3.1 and 3.2 are shown in Table 5. The variables and parameters included in the models are represented as ‘1’ and those not included in the models are left blank.

In all the five models, the effects of ‘Private English School’ α₂, ‘Tutor’ α₃ and ‘Favour’ α₉ were modelled in the parameter δ_i as fixed effects. The distinct feature of Model (T) modelled the interactions between the effect of ‘Gender’ and ‘Time’ α_1j, ‘Tool’ α_1k, ‘Time × Tool’ α_1jk respectively. In addition, the effect of ‘Frequency’ α_10jk affects Model (T). The distinct feature of Model (R) modelled the interactions between the effects of the children’s backgrounds, such as ‘Parents’ (α_5j, α_5k, α_5jk), ‘Home Learning’ (α_7j, α_7k, α_7jk), and ‘Homework of Private English School’ (α_8j, α_8k, α_8jk), and ‘Time’, ‘Tool’, ‘Time × Tool’ respectively. The effects of Model (W) were fewest compared to all the other models. As can be seen in Table 1, it seems that Model (W) became the simplest model of all the models since the level of writing in the textbook was only PreA1. The distinct features of Model (L) and Model (S) modelled the interactions between the effects of ‘Private English School’ and ‘Time’ α_2j, ‘Tool’ α_2k, ‘Time × Tool’ α_2jk respectively. It is noted that the interactions (α_1j, α_1k, α_1jk) between ‘Gender’ were modelled in Model (L) and Model(W), while these interactions were not modelled in Model (S).

The interaction (8.781) of ‘Time (j = 2) × Tool (k = 2: speaking-pen)’ was significantly positive. The results show that the children who learned using a speaking-pen during the first period were, on average, able to improve their own overall ability more than the children who used an audio CD. It seems that the speaking-pen is more effective than an audio CD in the early stage of English learning.

 In terms of variances, the estimated value of the variance of ‘Individual Differences’ (283.92) was larger than ‘Error’(86.42). It seems that the individual difference was large as a factor of a variance in Model (T).