nach oben

Educational Assessment, Evaluation and Accountability

Erschienen in:

01.02.2010

Mixing methods in randomized controlled trials (RCTs): Validation, contextualization, triangulation, and control

verfasst von: James P. Spillane, Amber Stitziel Pareja, Lisa Dorner, Carol Barnes, Henry May, Jason Huff, Eric Camburn

Erschienen in: Educational Assessment, Evaluation and Accountability | Ausgabe 1/2010

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

In this paper we described how we mixed research approaches in a Randomized Control Trial (RCT) of a school principal professional development program. Using examples from our study we illustrate how combining qualitative and quantitative data can address some key challenges from validating instruments and measures of mediator variables to examining how contextual factors interact with the treatment. Describing how we transformed our qualitative and quantitative data, we consider how mixing methods enabled us to deal with the two core RCT challenges of random assignment and treatment control critical. Our account offers insights into ways of maximizing the potential of mixing research methods in RCTs.

Vorheriger Artikel Editorial

Nächster Artikel Formative assessment systems: evaluating the fit between school districts’ needs and assessment systems’ characteristics

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

Cloverville is a pseudonym.

Birckmayer, J. D., & Weiss, C. H. (2000). Theory-based evaluation in practice: what do we learn? Evaluation Review, 24(4), 407–431.CrossRef

Bloom, H. S. (Ed.). (2005). Learning more from social experiments: Evolving analytic approaches. New York: Russell Sage Foundation.

Borman, G. D., Slavin, R. E., Cheun, A., Chamberlain, A. M., Madden, N. A., & Chambergs, B. (2005). Success for all: first-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1–22.CrossRef

Boruch, R. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks: Sage.

Boruch, R. (2002). The virtues of randomness. Education Next, 2(3), 36–41.

Brewer, J., & Hunter, A. (1989). Multimethod research: A synthesis of styles. Thousand Oaks: Sage.

Bryman, A. (2006). Integrating quantitative and qualitative research: how is it done? Qualitative Research, 6(1), 97–113.CrossRef

Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15(2), 195–207.

Caracelli, V. J., & Greene, J. C. (1997). Crafting mixed-method evaluation designs. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed methods evaluation: The challenges and benefits of integrating diverse paradigms. San Francisco: Jossey-Bass.

Chatterji, M. (2005). Evidence on "what works": an argument for extended-term mixed-method (ETMM) evaluation designs. Educational Researcher, 34(5), 14–24.CrossRef

Chen, H. T. (1990). Theory-driven evaluations. Newbury Park: Sage.

Chen, H. T. (1997a). Applying mixed methods under the framwork of theory-driven evaluations. New Directions for Program Evaluation, 1997(74), 61–72.CrossRef

Chen, H. T. (1997b). Normative evaluation of an anti-drug abuse program. Evaluation and Program Planning, 20(2), 195–204.CrossRef

Chen, H. T. (2005). Practical program evaluation: Assessing and improving planning, implementation and effectiveness. Thousand Oaks: Sage.

Chen, H. T., & Rossi, P. H. (1980). The multi-goal, theory-driven approach to evaluation: a model linking basic and applied social science. Social Forces, 59(1), 106–122.CrossRef

Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense. Evaluation Review, 7(3), 283–302.CrossRef

Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: a practical guide. The Journal of the Learning Sciences, 6(3), 271–315.CrossRef

Cook, T. D. (2002). Randomized experiments in educational policy research: a critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24(3), 175–199.CrossRef

Cook, T. D., & Reichardt, C. S. (Eds.). (1979). Qualitative and quantitative methods in evaluation research. Thousand Oaks: Sage.

Cook, T. D., & Wong, V. (2006). The IES agenda to institutionalize randomized clinical trials in educational research: Description and commentary. Paper presented at the Institute for Policy Research Spring 2006 Colloquium.

Cook, T. D., Murphy, R. F., & Hunt, H. D. (2000). Comer's school development program in Chicago: a theory-based evaluation. American Educational Research Journal, 37(2), 535–597.

Creswell, J. W. (2002). Educational research: Planning conducting, and evaluation quantitative and qualitative research. Upper Saddle River: Merrill Prentice Hall.

Creswell, J. W., Shope, R., Plano Clark, V. L., & Green, D. O. (2006). How interpretive qualitative research extends mixed methods research. Research in the Schools, 13(1), 1–11.

Cronbach, L. J. (1988). Five perspecitves on the validity argument. In H. Wainer & H. I. Braun (Eds.), Test validity. Hillsdale: Erlbaum.

Cronbach, L. J. (1989). Construct validation after thirty years. In R. L. Linn (Ed.), Intelligence: Measurement, theory, and public policy. Urbana: University of Illinois Press.

Datta, L. (1994). Paradigm wars: a basis for peaceful coexistence and beyond. New Directions for Program Evaluation, 61, 53–70.CrossRef

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). New York: McGraw-Hill.

Denzin, N. K. (1989). The research act: A theoretical introduction to sociological methods (3rd ed.). Englewood Cliffs: Prentice-Hall.

Denzin, N. K., & Lincoln, Y. S. (Eds.). (2005). The SAGE handbook of qualitative research (3rd ed.). Thousand Oaks: Sage.

Eisenhart, M., & Towne, L. (2003). Contestation and change in national policy on "scientifically based" education research. Educational Researcher, 32(7), 31–38.CrossRef

Flemming, K., Adamson, J., & Atkin, K. (2008). Improving the effectiveness of interventions in palliative care: the potential role of qualitative research in enhancing evidence from randomized controlled trials. Palliative Medicine, 22(2), 123–131.CrossRef

Goldring, E., Huff, J., Spillane, J. P., & Barnes, C. A. (2009). Measuring the learning-centered leadership expertise of school principals. Leadership and Policy in Schools, 8(2), 197–228.CrossRef

Gottlieb, N. H., Lovato, C. Y., Weinstein, R., Green, L. W., & Eriksen, M. P. (1992). The implementation of a restrictive worksite smoking policy in a large decentralized organization. Health Education & Behavior, 19(1), 77–100.CrossRef

Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93–98.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274.

Guba, E. G. (1990). The alternative paradigm dialog. In E. G. Guba (Ed.), The paradigm dialog. Newbury Park: Sage.

Hall, B., & Howard, K. (2008). A synergistic approach: conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2(3), 248–269.CrossRef

Howe, K. R. (1988). Against the quantitative-qualitative incompatibility thesis or dogmas die hard. Educational Researcher, 17(8), 10–16.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10(1), 42–61.CrossRef

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educational Researcher, 33(7), 14–26.CrossRef

Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462–472.CrossRef

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.CrossRef

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis. American Psychologist, 48(12), 1181–1209.CrossRef

Lynch, S., Szesze, M., Pyke, C., & Kuipers, J. (2007). Scaling up highly rated middle science curriculum units for diverse student populations: Features that affect collaborative research and vice versa. In B. Schneider & S.-K. McDonald (Eds.), Scale-up in education: Issues in practice (Vol. 2). Plymouth: Rowman & Litlefield.

Mactavish, J. B., & Schleien, S. J. (2004). Re-injecting spontaneity and balance in family life: parents' perspectives on recreation in families that include children with developmental disability. Journal of Intellectual Disability Research, 48(2), 123–141.CrossRef

Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13–17.

Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage.

Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. I. Braun (Eds.), Test validity. Hillsdale: Erlbaum.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousands Oaks: Sage.

Morgan, D. L. (1998). Practical strategies for combining qualitative and quantitative methods: applications to health research. Qualitative Health Research, 8, 362–376.CrossRef

Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40(2), 120–123.CrossRef

Morse, J. M. (2003). Principles of mixed methods and multimethod research design. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage.

Moss, P. A. (1992). Shifting conceptions of validity in educational measurement: implications for performance assessment. Review of Educational Research, 62(3), 229–258.

Moss, P. A. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5–12.

Nastasi, B. K., & Schensul, S. L. (2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43(3), 177–195.CrossRef

National Policy Board for Educational Administration (2002). Standards for advanced programs in educational leadership for principals, superintendents, curriculum directors and supervisors.

Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research. Tallinn: Tallinn Pedagogical University.

Petrosino, A. (2000). Answering the "why" question in evaluation: the causal-model approach. Canadian Journal of Program Evaluation, 15(1), 1–24.

Ragin, C. C. (2000). Fuzzy-set social science. Chicago: University of Chicago Press.

Reichardt, C. S., & Rallis, S. F. (Eds.). (1994). The qualitative-quantitative debate: New perspectives. San Francisco: Jossey-Bass.

Riggin, L. J. C. (1997). Advances in mixed-method evaluation: a synthesis and comment. New Directions for Program Evaluation, 74, 87–94.CrossRef

Rossi, P. H., Lipsey, M. W., & Freeman, D. J. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks: Sage.

Rossman, G. B., & Wilson, B. L. (1993). Numbers and words revisited: being "shamelessly eclectic". Quality and Quantity, 28(3), 315–327.CrossRef

Sandelowski, M. (1996). Focus on qualitative methods: using qualitative methods in intervention studies. Research in Nursing and Health, 19(4), 359–364.CrossRef

Sandelowski, M. (2003). Tables or tableaux? The challenges of writing and reading mixed methods studies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social dn behavioral research. Thousand Oaks: Sage.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park: Sage.

Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Academies Press.

Spillane, J. P., Camburn, E. M., & Pareja, A. S. (2007). Taking a distributed perspective to the school principal's workday. Leadership and Policy in Schools, 6(1), 103–125.CrossRef

Strauss, A. L., & Corbin, J. M. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Thousand Oaks: Sage.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage.

Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524.CrossRef

Yin, R. K. (2006). Mixed methods research: are the methods genuinely integrated or merely parallel? Research in the Schools, 13(1), 41–47.

Titel: Mixing methods in randomized controlled trials (RCTs): Validation, contextualization, triangulation, and control
verfasst von: James P. Spillane
Amber Stitziel Pareja
Lisa Dorner
Carol Barnes
Henry May
Jason Huff
Eric Camburn
Publikationsdatum: 01.02.2010
Verlag: Springer Netherlands
Erschienen in: Educational Assessment, Evaluation and Accountability / Ausgabe 1/2010
Print ISSN: 1874-8597
Elektronische ISSN: 1874-8600
DOI: https://doi.org/10.1007/s11092-009-9089-8

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 1/2010

The voice of the pupils: an experimental comparison of decisions made by elected pupil councils, pupils in referenda, and teaching staff

Erratum to: Why school based assessment is not a universal feature of high stakes assessment systems?

Formative assessment systems: evaluating the fit between school districts’ needs and assessment systems’ characteristics

Implementing the mandate: the limitations of benchmark tests

Editorial

Premium Partner