nach oben

Technology, Knowledge and Learning

Erschienen in:

09.11.2023 | Original research

A Road Less Traveled for Designing Rigorous Studies to Identify Effective Technology-Based Reading Interventions

verfasst von: Herb Turner, Raifu Durodoye

Erschienen in: Technology, Knowledge and Learning | Ausgabe 3/2024

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Designing rigorous studies to identify technology-based interventions to improve K-12 student reading skills require intervention researchers to take a “road less traveled.” This chapter presents that road using the Knowledge Acquisition and Transformation Expansions (KATE) intervention as a case study. An early version of KATE has shown positive, meaningful, and statistically significant effects on five dimensions of student reading comprehension skills in previous rigorous research with diverse samples. As a result, a new rigorous study of KATE is being designed to answer the question, can KATE replicate or improve on the positive effects in previous research at scale in 64 schools, 384 teachers, and 8448 students across multiple districts and states? Accurately answering this question requires thoughtful planning, designing, and recruiting with attention to Evidence Synthesis Clearinghouses review standards. Study results may be reviewed, rated, and publicly disseminated by these Clearinghouses beyond just peer-review journals.

Vorheriger Artikel Teachers’ Beliefs and Usage of Video Exemplars and Engagement Features of an Online Professional Learning System for Promoting Early Writing

Nächster Artikel An Analysis of Student Skills and Teacher Practices Across Writing Textbooks in Grades 4 and 5

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

This emphasis on the offer is called the “intent-to-treat” dimension of the RCT framework.

We reasoned that Peer Reviewers for EIR and academic journal would also view our causal interpretations of KATE more favorably if random assignment rather than non-random assignment was used.

The operative word is “potential.” There is valid previous evidence of effectiveness of KAT. However, whether this evidence can be replicated with KATE is an open empirical question that will be addressed by the proposed rigorous study of scaling KAT to KATE.

More technically called statistical power.

Confounds make it impossible to differentiate between the effect of the intervention and the effect of the confound on an outcome.

Statistically speaking the hierarchical design is less powerful than the randomized block design, ceteris paribus.

Whenever the unit of assignment is different from the unit of analysis, the analysis must account for this difference to calculate the correct standard errors for the impact estimate (Raudenbush and Bryk, 2002).

When study units have an equal probability of being assigned to each condition, the investigator need not account for it in the analysis (WWC, 2022).

This equating is a long-run statement about many replications of the same RCT. In any one RCT, the groups could differ on a pre-KATE characteristics; however, this difference would be due to chance and not systematic (Boruch, 1997).

The control group is a “wait-list” control that will receive access to KATE the following school year when KATE implementation has ended.

See WWC standards briefs at https://ies.ed.gov/ncee/wwc/Protocols#supplement.

See “Fig. 7. Ratings flowchart for cluster-level assignment studies” in WWC Procedures and Standards Handbook 5.0 (WWC 2022).

This is why in Fig. 12 of Handbook 5.0 (WWC, 2022), if a study uses an acceptable approach to missing data and attrition is low, ceteris paribus, the study “Meets WWC Group Design Standards without Reservations.”.

We recognize that missing data can be caused by sample exclusions based on exogenous factors outside the intervention researcher’s control like the COVID-19 pandemic. However, the WWC does not count these factors as attrition (Handbook 5.0, p.41).

For the WWC, “key” is defined by the Study Review Protocol and typically is a pre-intervention measure of the outcome. Otherwise, two or more are required such as SES and a demographic.

To examine impact on state standardized tests the research team will transform raw scores into z-scores. In analyses of impact on state standardized tests, the research team will also include a state-specific fixed effect in the impact model to control for unobserved heterogeneity across states.

Charter schools, magnet schools, vocational schools, and alternative schools are excluded from population frame.

Allison, P. D. (2000). Multiple imputation for missing data: A cautionary tale. Sociological Methods & Research, 28(3), 301–309. https://doi.org/10.1177/0049124100028003003CrossRef

Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48(1), 5–37.CrossRef

Bloom, H. S. (2005). Learning more from social experiments: Evolving analytic approaches. Russell Sage Foundation.

Boruch, R. F. (1997). Randomized experiments for planning and evaluation: A practical guide. Sage Publications.CrossRef

Boruch, R. F., & Turner, H. M. (2023). Randomized field experiments: Advances in practice. In Evaluation roots, third edition theory influencing practice. Guilford Publications.

Bosch, G., & Casadevall, A. (2017). Graduate biomedical science education needs a new philosophy. Mbio, 8(6), e01539-e1617. https://doi.org/10.1128/mBio.01539-17CrossRef

Boulay, B., Goodson, B., Olsen, R., McCormick, R., Darrow, C., Frye, M., Gan, K., Harvill, H., & Sarna, M. (2018). The investing in innovation fund: Summary of 67 evaluations: Final report (NCEE 2018–4013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Campbell, D. T., Stanley, J. C., & Gage, N. L. (1966). Experimental and quasi-experimental designs for research. Houghton Mifflin.

Campbell, M. J., & Walters, S. J. (2014). How to Design, Analyse and Report Cluster Randomised Trials in Medicine and Health Related Research. Wiley.CrossRef

Connolly, P., Keenan, C., & Urbanska, K. (2018). The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980–2016. Educational Research, 60, 1–16. https://doi.org/10.1080/00131881.2018.1493353CrossRef

De Smedt, F., Graham, S., & Van Keer, H. (2020). “It takes two”: The added value of structured peer-assisted writing in explicit writing instruction. Contemporary Educational Psychology. https://doi.org/10.1016/j.cedpsych.2019.101835CrossRef

Deke, J., & Chiang, H. (2017). The WWC attrition standard: Sensitivity to assumptions and opportunities for refining and adapting to new contexts. Evaluation Review, 41(2), 130–154. https://doi.org/10.1177/0193841X16670047CrossRef

Dignath, C., Büttner, G., & Langfeldt, H. P. (2007). The efficacy of self-regulated learning interventions at primary and secondary school level: A meta-analysis. In Paper presented at the EARLI biennial conference in Budapest, Hungary.

Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold.

Enders, C. K. (2022). Applied missing data analysis. Guilford Publications.

Granger, R., & Maynard, R. A. (2015). Unlocking the potential of the “what works” approach to policymaking and practice: Improving impact evaluations. American Journal of Evaluation, 36(4), 558–569. https://doi.org/10.1177/1098214015594420CrossRef

Gueron, J. (2002). The politics of random assignment: Implementing studies and affecting policy. In F. Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in education research (pp. 15–49). Brookings Institution Press. https://doi.org/10.7864/j.ctvc16n69.5CrossRef

Gueron, J. (2008). The politics of random assignment: Implementing studies and impacting policy. Journal of Children’s Services, 3(1), 14–26.CrossRef

Hariton, E., & Locascio, J. J. (2018). Randomised controlled trials—The gold standard for effectiveness research. BJOG, 125(13), 1716. https://doi.org/10.1080/00131881.2018.1493353CrossRef

Hedges, L. V., & Rhoads, C. (2010). Statistical power analysis in education research. NCSER 2010–3006. National Center for Special Education Research.

Hudson, A. K., Owens, J. K., Moore, K. A., Lambright, K., & Wijekumar, K. (2022). “What is the main idea?”: Using text structure as a framework for accelerating strategic comprehension of text. The Reading Teacher. https://doi.org/10.1002/trtr.2016CrossRef

Ioannidis, J. P., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166–175.CrossRef

Kern, M. L., & Friedman, H. S. (2008). Early educational milestones as predictors of lifelong academic achievement, midlife adjustment, and longevity. Journal of Applied Developmental Psychology, 30(4), 419–430. https://doi.org/10.1016/j.appdev.2008.12.025CrossRef

Lyon, G. R. (2001). Measuring success: Using assessments and accountability to raise student achievement. In Hearing before the Subcommittee on Education Reform, Committee on Education and the Workforce, U.S. House of Representatives, March 8, 2001.

Lyon, G. R., Shaywitz, S. E., Shaywitz, B. A., Chhabra, V., & Adams, M. J. (2005). Evidence-based reading policy in the United States: How scientific research informs instructional practices. Brookings papers on education policy, 8, 209–250.CrossRef

Maynard, R. A. (2018). The role of federal agencies in creating and administering evidence-based policies. The ANNALS of the American Academy of Political and Social Science, 678(1), 134–144. https://doi.org/10.1177/0002716218768742CrossRef

Maynard, R., Goldstein, N., & Nightingale, D. S. (2016). Program and policy evaluations in practice: Highlights from the federal perspective. In L. Peck (Ed.), New directions for evaluation (pp. 109–134). Jossey-Bass/Wiley. https://doi.org/10.1002/ev.20209CrossRef

Meyer, B. J. F. (1975). The organization of prose and its effects on memory. North-Holland.

Meyer, B. J. F., & Poon, L. W. (2001). Effects of structure strategy training and signaling on recall of text. Journal of Educational Psychology, 93, 141–159. https://doi.org/10.1037/0022-0663.93.1.141CrossRef

Meyer, B. J. F., & Wijekumar, K. (2007). A web-based tutoring system for the structure strategy: Theoretical background, design, and findings. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 347–375). Lawrence Erlbaum Associates.

Msila, V., & Setlhako, A. (2013). Evaluation of Programs: Reading Carol H. Weiss. Universal Journal of Educational Research, 1(4), 323–327.CrossRef

National Research Council. (2002). Scientific research in education. National Academies Press.

National Research Council. (2004). Advancing scientific research in education. National Academies Press.

National Assessment of Educational Progress. (2019). The nation’s report card: Reading 2017.

Orr, L. L. (1999). Social experiments: Evaluating public programs with experimental methods. Sage Publications.

Puma, Michael J., Olsen, R. B, Bell, S. H., & Price, C. (2009). What to do when data are missing in group randomized controlled trials (NCEE 2009–0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Pyle, N., Vasquez, A. C., Lignugaris Kraft, B., Gillam, S. L., Reutzel, D. R., Olszewski, A., Segura, H., Hartzheim, D., Laing, W., & Pyle, D. (2017). Effects of expository text structure interventions on comprehension: A meta-analysis. Reading Research Quarterly, 52, 469–501. https://doi.org/10.1002/rrq.179CrossRef

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Sage Publications.

Reardon, K. (2011). To what degree will learning to use context clues impact students’ reading comprehension scores? [Doctoral dissertation, University of Maryland]. ProQuest Dissertations and Theses Global.

Research to Policy Collaboration. (2020). Clearinghouses and evidence-based resources. Retrieved January 30, 2023, from https://research2policy.org/clearinghouses-and-evidence-based-resources/

Rhoads, C. H. (2011). The implications of “contamination” for experimental design in education. Journal of Educational and Behavioral Statistics, 36(1), 76–104. https://doi.org/10.3102/1076998610379133CrossRef

Rhoads, C. (2016). The implications of contamination for educational experiments with two levels of nesting. Journal of Research on Educational Effectiveness, 9(4), 531–555.CrossRef

Ritchie, S. J., & Bates, T. C. (2013). Enduring links from childhood mathematics and reading achievement to adult socioeconomic status. Psychological Science, 24(7), 1301–1308. https://doi.org/10.1177/0956797612466268CrossRef

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350CrossRef

Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics. https://doi.org/10.1214/08-aoas187CrossRef

Sahni, S. D., Polanin, J. R., Zhang, Q., Michaelson, L. E., Caverly, S., Polese, M. L., & Yang, J. (2021). A what works clearinghouse rapid evidence review of distance learning programs. WWC 2021–005REV. What Works Clearinghouse.

Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming. In A. W. Siegman & S. Feldstein (Eds.), Advances in applied psycholinguistics: Volume 2, Reading, writing, and language learning (pp. 142–175). Ablex.

Schochet, P. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87. https://doi.org/10.3102/1076998607302714CrossRef

Schochet, P. Z. (2009). An Approach for addressing the multiple testing problem in social policy impact evaluations. Evaluation Review, 33(6), 539–567. https://doi.org/10.1177/0193841x09350590CrossRef

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Song, M., & Herman, R. (2010). Critical issues and common pitfalls in designing and conducting impact studies in education: Lessons learned from the what works clearinghouse (Phase I). Educational Evaluation and Policy Analysis, 32(3), 351–371. https://doi.org/10.3102/0162373710373389CrossRef

Song, M., Wayne, A. J., Garet, M. S., Brown, S., & Rickles, J. (2021). Impact of providing teachers and principals with performance feedback on their practice and student achievement: Evidence from a large-scale randomized experiment. Journal of Research on Educational Effectiveness, 14(2), 353–378.CrossRef

Spybrook, J., Raudenbush, S. W., Liu, X. F., Congdon, R., & Martínez, A. (2006). Optimal design for longitudinal and multi-level research: Documentation for the “Optimal Design” software. Survey Research Center of the Institute of Social Research at University of Michigan.

Spybrook, J., & Raudenbush, S. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the institute of education sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318. https://doi.org/10.3102/0162373709339524CrossRef

Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences. International Journal of Research and Method in Education, 39(3), 255–267. https://doi.org/10.1080/1743727X.2016.1187914CrossRef

Texas Education Agency. (2018). STAAR® Grades 3-8 Reading Released Test Forms and Answer Keys [PDF file]. Retrieved from https://tea.texas.gov/student-assessment/testing/staar/staar-releasedtest/2018staargr3readingtest.pdf

Tipton, E., & Olsen, R. B. (2022). Enhancing the Generalizability of Impact Studies in Education. Toolkit. NCEE 2022-003. National Center for Education Evaluation and Regional Assistance.

Tipton, E. & Miller, K. (2022) Generalizer [Web-tool]. Retrieved at https://thegeneralizer.org.

Tipton, E., & Hedges, L. V. (2017). The role of the sample in estimating and explaining treatment effect variation: A commentary on three papers. Journal of Research on Educational Effectiveness., 10(4), 903–906. https://doi.org/10.1080/19345747.2017.1361468CrossRef

Tipton, E., & Matlen, B. (2019). The development and implementation of a strategic recruitment plan for generalizability: A case study. American Journal of Evaluation, 40(3), 414–430.CrossRef

Tipton, E., & Olsen, R. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. https://doi.org/10.3102/0013189X18781522CrossRef

U.S. Department of Education, Institute of Education Sciences. (2018). Standards for excellence in education research. https://ies.ed.gov/seer.as

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. (2017). Procedures and standards handbook (Version 4.1) [PDF file]. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v4_1_standards_handbook.pdf

Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? The American Journal of Evaluation, 19(1), 21–33.CrossRef

Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719CrossRef

Wijekumar, K. K., Meyer, B. J. F., & Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Educational Technology Research and Development, 60(6), 987–1013.CrossRef

Wijekumar, K., Meyer, B. J. F., & Lei, P. (2017). Web-based text structure strategy instruction improves seventh graders’ content area reading comprehension. Journal of Educational Psychology, 109(6), 741–760.CrossRef

Wijekumar, K., Meyer, B. J. F., Lei, P., Beerwinkle, A. L., & Joshi, M. (2020). Supplementing teacher knowledge using web-based Intelligent Tutoring System for the Text Structure Strategy to improve content area reading comprehension with fourth- and fifth-grade struggling readers. Dyslexia, 26, 120–136. https://doi.org/10.1002/dys.1634CrossRef

What Works Clearinghouse (WWC) (2022). What Works Clearinghouse Procedures and Standards Handbook, Version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance (NCEE). Retrieved from https://ies.ed.gov/ncee/wwc/Handbooks

Zhang, Q., Spybrook, J., & Unlu, F. (2020). Examining design and statistical power for planning cluster randomized trials aimed at improving student science achievement and science teacher outcomes. AERA Open. https://doi.org/10.1177/2332858420939526CrossRef

Titel: A Road Less Traveled for Designing Rigorous Studies to Identify Effective Technology-Based Reading Interventions
verfasst von: Herb Turner
Raifu Durodoye
Publikationsdatum: 09.11.2023
Verlag: Springer Netherlands
Erschienen in: Technology, Knowledge and Learning / Ausgabe 3/2024
Print ISSN: 2211-1662
Elektronische ISSN: 2211-1670
DOI: https://doi.org/10.1007/s10758-023-09695-1

Instructional Support for Visual Displays: An Updated Literature Review

Original research

Developing a Scale of Attitudes Towards Flipped Learning Model

Original research

The Effect of a Teacher’s Mindset on the Cascading Zones of Proximal Development: A Systematic Review

Original research

Parent and Family Support of Knowledge Acquisition in Reading Comprehension through Podcasts in English and Spanish

Original research

Digital Teacher Training in the Portuguese National Plan for Digital Development at Schools: A Case Study

Open Access
Original research

Voluntary E-Learning Exercises Support Students in Mastering Statistics

Open Access
Original research

Premium Partner

Marktübersichten

Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.

Zur Marktübersicht

Bildnachweise

MKVS GbR/© MKVS GbR, Nordson/© Nordson, ViscoTec/© ViscoTec, BCD Chemie GmbH, Merz+Benteli/© Merz+Benteli, Robatech/© Robatech, Hermann Otto GmbH/© Hermann Otto GmbH, Ruderer Klebetechnik GmbH, Xometry Europe GmbH/© Xometry Europe GmbH, Scheugenpflug/© Scheugenpflug, Sika/© Sika, Medmix/© Medmix, Kisling AG/© Kisling AG, Dosmatix GmbH/© Dosmatix GmbH, Innotech GmbH/© Innotech GmbH, Hilger u. Kern GmbH, VDI Logo/© VDI Wissensforum GmbH, Dr. Fritz Faulhaber GmbH & Co. KG/© Dr. Fritz Faulhaber GmbH & Co. KG, BEINLICH PUMPEN GMBH/© BEINLICH PUMPEN GMBH