Skip to main content
SpringerLink
Log in

Validity in quantitative content analysis

  • Research
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anderson, T., Rourke, L., Garrison, D.R., & Archer, W. (2001) Assessing teaching presence in a computer conferencing environment.Journal of Asynchronous Learning Networks,5 (2). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/jalnvol5issue2v2.htm

  • Bakeman, R., & Gottman, J.M. (1997).Observing interaction: An introduction to sequential analysis (2nd ed.). New York: Cambridge University Press.

    Google Scholar 

  • Bereiter, C., & Scardemalia, M. (1987).The psychology of written composition. Hillsdale N.J.: Lawrence Erlbaum Associates.

    Google Scholar 

  • Berelson, B. (1952).Content analysis in communication research. Glencoe, IL: Free Press.

    Google Scholar 

  • Bullen, M. (1998). Participation and critical thinking in online university distance education.Journal of Distance Education, 13(2), 1–32.

    Google Scholar 

  • Chou, C. (November, 2001).A model of learner-centered computer-mediated interaction for collaborative distance learning. Paper presented at the Annual Meeting of the Association for Educational and Communications Technology, Atlanta, GA.

  • Community of Inquiry. (2002). Critical thinking in a text-based environment: Computer conferencing in higher education. Retrieved March 6, 2002, from http://www.atl.ualberta.ca/cmc

  • Grocker, L., & Algina, J. (1986).Introduction to classical and modern test theory. Toronto: Harcourt Brace Jovanovich College Publishers.

    Google Scholar 

  • Cronbach, L. (1971). Test validation. In R.L. Thorndike (Ed.),Educational Measurement (2nd ed.). Washington, D.C.: American Council on Education.

    Google Scholar 

  • Cronbach, L. (1990).Essentials of psychological testing (5rd ed.). New York: Harper & Row.

    Google Scholar 

  • Curtis, D., & Lawson M. (2001) Exploring collaborative learning online.Journal of Asynchronous Learning Networks,5(1). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/Vol5_issue1/Curtis/curtis.htm

  • Dalkey, N., & Helmer, O. (1963). An experimental application of the delphi method to the user of experts.Management Science, 9(3), 458–467.

    Article  Google Scholar 

  • Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span.Psychological Bulletin, 93, 179–197.

    Article  Google Scholar 

  • Ericsson, K., & Simon, H. (1993)Protocol analysis: Verbal reports as data. Cambridge, Mass: MIT Press.

    Google Scholar 

  • Fahy, P. (2001). Addressing some common problems in transcript analysis.International Review of Research in Open and Distance Learning,1(2). Retrieved, March 20, 2002, from the World Wide Web at. http://www.irrodl.org/content/v1.2/research.html

  • Fahy, P. (2002a). Epistolary and expository interactions patterns in a computer conference transcript. Retrieved, March 1, 2002, from http://cde.athabascau.ca/softeval/reports/mag2-jde.pdf

  • Fahy, P. (2002b). Evaluating critical thinking in a com

  • Fahy, P. (in press). Use of linguistic qualifiers and intensifiers in a computer conference.American Journal of Distance Education.

  • Flanagan, J. (1954). The critical incident technique.Psychological Bulletin, 51(4), 327–359.

    Article  Google Scholar 

  • Gall, M., Borg, W., & Gall, J. (1996).Educational research: An introduction (6th ed.). White Plains, NY: Longman.

    Google Scholar 

  • Garrison, D.R., Anderson, T., & Archer, W. (2001). Critical thinking, congitive presence, and computer conferencing in distance education.American Journal of Distance Education,15(1).

  • Gunawardena, C.N., Lowe, C.A., & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing.Journal of Educational Computing Research 17(4), 397–431.

    Google Scholar 

  • Gunawardena, C.N., & Zittle, F. (1998). Social presence as a predictor of satisfaction within a computer mediated conferencing environment.The American Journal of Distance Education, 11(3), 8–25.

    Article  Google Scholar 

  • Hara, N., Bonk, C. & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course.Instructional Science, 28(2), 115–152.

    Article  Google Scholar 

  • Henri, F. (1991). Computer conferencing and content analysis. InCollaborative learning through computer conferencing (pp. 117–136). Berlin: Springer-Verlag.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1979). Conflict in the classroom: Controversy and learning.Review of Educational Research, 49, 51–70.

    Article  Google Scholar 

  • Johnson, D., & Johnson, R. (1986). Computer-assisted cooperative learning.Educational Technology, 26(1), 12–18.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1989).Cooperation and competition: theory and research. Edina, MN: Interaction.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1992a).Creative controversy: Intellectual challenge in the classroom. Edina, MN: Interaction.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1992b). Positive interdependence: key to effective cooperation. In R. Hertz-Lazarowitz & N. Miller (Eds.),Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 174–99). Cambridge, England: Cambridge University Press.

    Google Scholar 

  • Johnson, D.W., & Johnson, F. (1994a).Joining together. Group theory and group skills (5th ed). Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1994b).Leading the cooperative school (2nd ed.) Edina, MN: Interaction.

    Google Scholar 

  • Johnson, D., & Johnson, R. (1996). Cooperation and the use of technology. In D. Jonassen (Ed.),Handbook of research for educational communications and technology (pp. 1017–1044). New York: Simon and Schuster Macmillan. (1996).

    Google Scholar 

  • Johnson, D.W., Johnson, R., & Maruyama, G. (1983). Interdependence and interpersonal attraction among heterogeneous and homogeneous individuals: A theoretical formation and a meta-analysis of the research.Review of Educational Research, 53(5), 5–54.

    Article  Google Scholar 

  • Johnson, D., Johnson, R., & Stanne, M. (2000).Cooperative learning methods: A meta-analysis. Retrieved, March 6, 2003, from http://www.co-operation.org/pages/cl-methods.html

  • Jonassen, D., & Kwon, H. (2001). Communication patterns in computer mediated versus face-to-face group problem solving.Educational Technology Research and Development, 49(1), 35–51.

    Article  Google Scholar 

  • Jonassen, D., & Kwon, H. (2002). The effects of argumentation scaffolds on argumentation and problem solving.Educational Technology Research and Development, 50(3), 5–21.

    Article  Google Scholar 

  • Kamin, C., O'Sullivan, P., Younger, M., & Deterding, R. (2001). Measuring critical thinking in problem-based learning discourse.Teaching and Learning in Medicine, 13(1), 27–35.

    Article  Google Scholar 

  • Kaplan, A. (1964).The conduct of inquiry: Methodology for behavioral science. Scranton PA: Chandler.

    Google Scholar 

  • Krippendorff, K. (1980).Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.

    Google Scholar 

  • Lord, F., & Novick, M. (1968).Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Mason, R. (1991). Analyzing computer conference interactions.Computer in Adult Education and Training, 2(3), 161–173.

    Google Scholar 

  • McLean, S., & Morrison, D. (2000). Learners sociodemographic characteristics and participation in computer conferencing.Journal of Distance Education,15(2). Retrieved, March 9, 2002, from http://cade.athabascau.ca/vol15.2/mclean.html

  • Messick, S. (1989). Validity. In R.L. Linn (Ed.),Educational measurement (3rd ed, pp. 13–103). New York: Macmillan

    Google Scholar 

  • Norris, S., & Ennis, R. (1989).Evaluating critical thinking. CA: Critical Thinking Press and Software.

    Google Scholar 

  • Paisley, W. (1969). Studying style as deviation from encoding norms. In G. Gerbner, O. Holsti, K. Krippendorf, W. Paisley, & P. Stone (Eds.),The analysis of communication contents: Developments in scientific theories and computer techniques (pp. 4458). New York: Wiley.

    Google Scholar 

  • Parson, M. (1996). Look who's talking: A pilot study of the use of discussion lists by journalism educators and students. (ERIC Document Reproduction Service No. ED 400 562)

  • Poole, M., & Holmes, M. (1995). The longitudinal analysis of interaction. In B. Montgomery & S. Duck (Eds.),Studying interpersonal interaction (pp. 286–302). New York: Guilford. 1991.

    Google Scholar 

  • Potter, J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis.Journal of Applied Communication Research, 27(3): 258–284.

    Article  Google Scholar 

  • Reeves, T. (1995). Questioning the questions of instructional technology research. [Online] Available http: //www.hbg.psu.edu/bsed/intro/docs/dean/

  • Richardson, J., & Swan, K. (2003). Examining social presence in online courses in relation to students' perceived learning and satisfaction.Journal of Asynchronous Learning Networks 7(1). Retrieved July 1, 2003 from the World Wide Web at http://www.aln.org/publications/jaln/v7n1/v7n 1_richardson.asp

  • Riffe, D., Lacy, S., & Fico, F. (1998)Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Rogers, W.T. (1999).Error of measurement and validity. Edmonton AB: Available from author.

    Google Scholar 

  • Rourke, L., & Anderson, T. (2002). Social communication in computer conferencing.Journal of Interactive Learning Research, 13(3), 259–275.

    Google Scholar 

  • Rourke, L., Anderson, T. Garrison, D.R., & Archer, W. (1999). Assessing social presence in asynchronous, text-based computer conferencing.Journal of Distance Education, 14(3), 51–70.

    Google Scholar 

  • Rourke, L., Anderson, T., Garrison, D.R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts.International Journal of Artificial Intelligence in Education, 12(1), 8–22.

    Google Scholar 

  • Sharon, Y., & Sharon, S. (1992).Group investigation: Expanding cooperative learning. New York: Teacher's College Press.

    Google Scholar 

  • Sheppard, L. (1993). Evaluating test validity.Review of Research in Education, 19, 405–450.

    Article  Google Scholar 

  • Short, J., Williams, E., & Christie, B. (1976).The social psychology of telecommunications. London, U.K.: Wiley, 1976.

    Google Scholar 

  • Slavin, R. (1991).Student team learning: A practical guide to cooperative learning (3rd ed.). Washington, DC: National Education Association.

    Google Scholar 

  • Snow, R., Federico, P., & Montague, W. (Eds.). (1980).Aptitude, learning, and instruction. (Vols. 1 & 2). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Sutton, L. (2001). The principle of vicarious interaction in computer mediated communication.International Journal of Educational Telecommunications, 7(3), 223–242.

    Google Scholar 

  • Stevens, S. (1946). On the theory of scales of measurement.Science, 103, 677–680.

    Article  Google Scholar 

  • Weiss, R., & Morrison, G. (1998). Evaluation of a graduate seminar conducted by listserve. (ERIC Document Reproduction Service No. ED 423 868)

Download references

Author information

Authors and Affiliations

  1. Department of Educational Psychology at the University of Alberta, Edmonton, AB

    Liam Rourke Ph.D.

  2. Athabasca University in Alberta, Alberta, Canada

    Terry Anderson (Professor and Canada Research Chair in Distance Education)

Authors
  1. Liam Rourke Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Terry Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rourke, L., Anderson, T. Validity in quantitative content analysis. ETR&D 52, 5–18 (2004). https://doi.org/10.1007/BF02504769

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02504769

Keywords

Search

Navigation