Elsevier

Educational Research Review

Volume 7, Issue 3, December 2012, Pages 198-215
Educational Research Review

Review
How features of educational technology applications affect student reading outcomes: A meta-analysis

https://doi.org/10.1016/j.edurev.2012.05.002Get rights and content

Abstract

The purpose of this review is to learn from rigorous evaluations of alternative technology applications how features of using technology programs and characteristics of their evaluations affect reading outcomes for students in grades K-12. The review applies consistent inclusion standards to focus on studies that met high methodological standards. A total of 84 qualifying studies based on over 60,000 K-12 participants were included in the final analysis. Consistent with previous reviews of similar focus, the findings suggest that educational technology applications generally produced a positive, though small, effect (ES = +0.16) in comparison to traditional methods. There were differential impacts of various types of educational technology applications. In particular, the types of supplementary computer-assisted instruction programs that have dominated the classroom use of educational technology in the past few decades were not found to produce educationally meaningful effects in reading for K-12 students (ES = +0.11), and the higher the methodological quality of the studies, the lower the effect size. In contrast, innovative technology applications and integrated literacy interventions with the support of extensive professional development showed more promising evidence. Although many more rigorous, especially randomized, studies of newer applications are needed, what unifies the methods found in this review to have great promise is the use of technologies in close connection with teachers’ efforts.

Highlights

► The findings suggest that educational technology produced a positive, though small, effect. ► There were differential impacts of various types of educational technology applications. ► Traditional computer-assisted instruction programs produced a small effect. ► Applications with the support of professional development showed more promising evidence.

Introduction

The classroom use of educational technology such as computers, interactive whiteboards, multimedia, and the internet, has been growing at a phenomenal rate in the last two decades. According to a recent survey conducted by the U.S. Department of Education (SETDA, 2010) on the use of educational technology in U.S. public schools, almost all public schools had one or more instructional computers with internet access, and the ratio of students to instructional computers with internet access was 3.1–1. In addition, 97% of schools had one or more instructional computers located in classrooms and 58% of schools had laptops on carts. A majority of public schools surveyed also indicated their schools provided various educational technology devices for instruction: LCD (liquid crystal display) and DLP (digital light processing) projectors (97%), digital cameras (93%), and interactive whiteboards (73%). The U.S. Department of Education provides generous grants to state education agencies to support the use of educational technology in K-12 classrooms. For example, in fiscal year 2009, the Department made a $900 million investment in educational technology in elementary and secondary schools (SETDA, 2010).

The debate around the effectiveness of educational technology for improving student learning has been carried on for over three decades. Perhaps the most widely cited debate was between Clark (1983) and Kozma (1994). Clark (1983) first argued that educational technology had no impact on student learning under any condition and that “media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition”. He continued to argue that the impact of technology on student learning was mainly due to novelty effects or instructional strategies, but not technology itself. Kozma (1994) responded to Clark’s argument by saying the analogy of “delivery truck” creates an “unnecessary schism between medium and method”. Kozma believed that technology had an actual impact on student learning and played an important role in student learning.

The Clark–Kozma debate of the 1980’s has been overtaken by the extraordinary developments in technology applications in education in recent years. It may be theoretically interesting to ask whether the impact of technology itself can be separated from the impact of particular applications, but as a practical matter, machine and method are intertwined. As is the case for many educational interventions with many components, currently available technology applications can be seen as packages of diverse elements and evaluated as such. If a particular combination of hardware, software, print materials, professional development for teachers, and other elements can be reliably replicated in many classrooms, then it is worth evaluating as a potential means of enhancing student outcomes. Components of effective multi-element treatments can be varied to find out which elements contribute to effectiveness and to advance theory, but it is also of value for practice and policy to know the overall impact for students even if the theoretical mechanisms are not yet fully understood. Technology is here to stay, and pragmatically, the question is how to make the best use of the many technologies now available.

Section snippets

Research on educational technology applications

Research on the effectiveness of various forms of educational technology applications such as computer-assisted instruction (CAI) for improving learning outcomes has been abundant since the 1980s. Several major meta-analyses of the impact of educational technology on reading have also been conducted in the past two decades (Becker, 1992, Blok et al., 2002, Fletcher-Finn and Gravatt, 1995, Kulik and Kulik, 1991, Kulik, 2003, Ouyang, 1993, Soe et al., 2000). Overall, all came to a similar

Method

The current review employed meta-analytic techniques proposed by Glass, McGaw, and Smith (1981) and Lipsey and Wilson (2001). Comprehensive Meta-analysis Software Version 2 (Borenstein, Hedges, Higgins, & Rothstein, 2005) was used to calculate effect sizes and to carry out various meta-analytical tests, such as Q statistics and sensitivity analyses. Like many previous meta-analyses, this study follows several key steps: (1) locating all possible studies; (2) screening potential studies for

Overall effects

A total of 84 qualifying studies based on 60,553 K-12 participants were included in the final analysis: 8kindergarten studies (N = 2068), 59elementary studies (N = 34,200), and 18 secondary studies (N = 24,285). As indicated in Table 2, the overall mean effect size for the 84 qualifying studies is +0.16. The distribution of effect sizes in this collection of studies is highly heterogeneous (Q = 362.52, df = 83, p < 0.00), indicating that the variance of study effect sizes is larger than can be explained by

Limitations

It is important to mention several limitations in this review. First, due to the scope of this review, only studies with quantitative measures of reading were included. There is much to be learned from other non-experimental studies such as qualitative and correlational research that can add depth and insight to understanding the effects of these educational technology programs. Second, the review focuses on replicable programs used in realistic school settings over periods of at least 12 weeks,

Discussion

The purpose of this review was to examine the overall effectiveness of educational technology applications on reading outcomes in K-12 classrooms. Consistent with previous reviews of similar focus (Kulik, 2003, Kulik and Kulik, 1991, Soe et al., 2000), the findings of this study suggest that applications of educational technology generally produced a positive, though small, effect (ES = +0.16) in comparison to traditional methods. This effect is much larger than those reported in the recent

Practical implications for designing effective technology applications

The findings of this study have some practical implications for designing effective technology applications for reading. First, it is important to discuss the potential benefits and drawbacks of technology applications to illustrate how design of software and the human systems in which technology applications operate might determine reading outcomes. It may be impossible to determine the unique contribution of technology itself, but it should be possible to learn how to maximize technology’s

Conclusions

The findings of this review support those of earlier reviews by other researchers. The classroom use of educational technology will undoubtedly continue to expand and play an increasingly significant role in public education in the years to come as technology becomes more sophisticated and more cost-effective. This review highlights the need for more randomized studies. In addition, schools and districts should make concerted efforts to identify and adopt research-proven educational technology

References (128)

  • R.C. Atkinson et al.

    Teaching children to read with a computer

    The Reading Teacher

    (1972)
  • Barker, T. A., & Torgesen, J. K. (1995). An evaluation of computer-assisted instruction in phonological awareness with...
  • ∗Barnett, L. B. (2006). The effect of computer-assisted instruction on the reading skills of emergent readers....
  • G. Bass et al.

    Teaching basic skills through microcomputer assisted instruction

    Journal of Educational Computing Research

    (1986)
  • ∗Beasley, N. (1989). The effects of IBM Writing to Read program on the achievement of selected first grade students....
  • H.J. Becker

    Computer-based integrated learning systems in the elementary and middle grades: A critical review and synthesis of evaluation reports

    Journal of Educational Computing Research

    (1992)
  • M.C. Biggs et al.

    Using an interactive singing software program: A comparative study of struggling middle school readers

    Reading Psychology

    (2008)
  • ∗Birch, J. (2002). The effects of the Delaware Challenge Grant Program on the standardized reading and mathematics test...
  • ∗Block, C. C., Campbell, M. J., Ninon, K., Williams, C., Helgert, M. (2007). Effects of AWARD Reading, a...
  • H. Blok et al.

    Computer-assisted instruction in support of beginning reading instruction: A review

    Review of Educational Research

    (2002)
  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2005). Comprehensive meta-analysis (Version 2)....
  • M. Borenstein et al.

    Introduction to meta-analysis

    (2009)
  • ∗Bryg, V. (1984). The effect of computer assisted instruction upon reading achievement with selected fourth grade...
  • W. Bucleitner

    Why do kids love computers?

    Scholastic Early Childhood Today

    (1996)
  • ∗Caggiano, J. A. (2007). Addressing the learning needs of struggling adolescent readers: The impact of a reading...
  • ∗Campbell, J.P. (2000). A comparison of computerized and traditional instruction in the area of elementary reading....
  • ∗Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematics software...
  • J. Cassady et al.

    The impact of a structured integrated learning system on first grade students’ reading gains

    Reading and Writing Quarterly

    (2005)
  • B. Chambers et al.

    Achievement effects of embedded multimedia in a Success for all reading program

    Journal of Educational Psychology

    (2006)
  • B. Chambers et al.

    Small-group computer-assisted tutoring to improve reading outcomes for struggling first and second graders

    Elementary School Journal

    (2011)
  • B. Chambers et al.

    Technology infusion in success for all: Reading outcomes for first graders

    Elementary School Journal

    (2008)
  • ∗Clariana, R.B. (1994). The effects of an integrated learning system on third graders’ mathematics and reading...
  • R.E. Clark

    Reconsidering research on learning from media

    Review of Educational Research

    (1983)
  • R.E. Clark

    Confounding in education computing research

    Journal of Educational Computing Research

    (1985)
  • R.E. Clark

    Evidence for confounding in computer-based instruction studies: Analyzing the meta-analyses

    Educational Communication and Technology Journal

    (1985)
  • ∗Clayton, I.L. (1992). The relationship between computer-assisted instruction in reading and mathematics achievement...
  • B. Collis et al.

    Writing to read: An evaluation of a Canadian installation of a computer-supported initial language environment

    Journal of Educational Computing Research

    (1990)
  • ∗Coomes, P. (1985). The effects of computer assisted instruction on the development of reading and language skills....
  • ∗Cooperman, K.S. (1985). An experimental study to compare the effectiveness of a regular classroom reading program to a...
  • J. Davidson et al.

    Scholastic’s READ 180: A heritage of research

    (2002)
  • ∗Denson, K. (2008). Passport Reading Journeys Effectiveness with Ninth Grade Students Identified for Reading...
  • ∗DiLeo, J. (2007). A study of a specific language arts and mathematics software program: Is there a correlation between...
  • ∗Dynarski, M., Agodini, R., Heaviside, S. N., Carey, N., Campuzano, L., Means, B., et al. (2007). Effectiveness of...
  • ∗Easterling, B. (1982). The effects of computer assisted instruction as a supplement to classroom instruction in...
  • L.C. Ehri et al.

    Reading rescue: An effective tutoring intervention model for language-minority students who are struggling readers in first grade

    American Educational Research Journal

    (2007)
  • R. Erdner et al.

    The impact of a year of computer assisted instruction on the development of first grade reading skills

    Journal of Educational Computing Research

    (1997)
  • ∗Estep, S. (1997). An investigation of the relationship between integrated learning systems and academic achievement....
  • ∗Faux, R. (2004). Evaluation of Lexia Software in Boston Public Schools, Final Report. Davis Square Research...
  • C. Fletcher-Finn et al.

    The efficacy of computer-assisted instruction (CAI): A meta-analysis

    Journal of Educational Computing Research

    (1995)
  • J.A. Frechtling et al.

    Voyager universal literacy system: Results from a study of Kindergarten students in inner-city schools

    Journal of Education for Students Placed at Risk

    (2006)
  • Cited by (190)

    View all citing articles on Scopus
    1

    Tel.: +1 410 616 2310.

    View full text