Skip to main content Accessibility help
×
Hostname: page-component-848d4c4894-x24gv Total loading time: 0 Render date: 2024-05-14T14:23:52.758Z Has data issue: false hasContentIssue false

4 - Students as Experimental Participants

A Defense of the “Narrow Data Base”

Published online by Cambridge University Press:  05 June 2012

James N. Druckman
Affiliation:
Northwestern University
Cindy D. Kam
Affiliation:
Vanderbilt University
James N. Druckman
Affiliation:
Northwestern University, Illinois
Donald P. Greene
Affiliation:
Yale University, Connecticut
James H. Kuklinski
Affiliation:
University of Illinois, Urbana-Champaign
Arthur Lupia
Affiliation:
University of Michigan, Ann Arbor
Get access

Summary

An experiment entails randomly assigning participants to various conditions or manipulations. Given common consent requirements, this means experimenters need to recruit participants who, in essence, agree to be manipulated. The ensuing practical and ethical challenges of subject recruitment have led many researchers to rely on convenience samples of college students. For political scientists who put particular emphasis on generalizability, the use of student participants often constitutes a critical, and according to some reviewers, fatal problem for experimental studies.

In this chapter, we investigate the extent to which using students as experimental participants creates problems for causal inference. First, we discuss the impact of student subjects on a study's internal and external validity. In contrast to common claims, we argue that student subjects do not intrinsically pose a problem for a study's external validity. Second, we use simulations to identify situations when student subjects are likely to constrain experimental inferences. We show that such situations are relatively limited; any convenience sample poses a problem only when the size of an experimental treatment effect depends on a characteristic on which the convenience sample has virtually no variance. Third, we briefly survey empirical evidence that provides guidance on when researchers should be particularly attuned to taking steps to ensure appropriate generalizability from student subjects. We conclude with a discussion of the practical implications of our findings.

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, Craig A., and Bushman, Brad J.. 1997. “External Validity of ‘Trivial’ Experiments: The Case of Laboratory Aggression.” Review of General Psychology 1: 19–41.CrossRefGoogle Scholar
Ansolabehere, Stephen, and Iyengar, Shanto. 1995. Going Negative: How Political Advertisements Shrink and Polarize the Electorate. New York: Free Press.Google Scholar
Arceneaux, Kevin, and Johnson, Martin. 2008. “Choice, Attention, and Reception in Political Communication Research.” Paper presented at the annual meeting of the International Society for Political Psychology, Paris.
Aronson, Elliot, Brewer, Marilynn B., and Carlsmith, J. Merill. 1985. “Experimentation in Social Psychology.” In Handbook of Social Psychology. 3rd ed., eds. Lindzey, Gardner and Aronson, Elliot. New York: Random House, 441–86.Google Scholar
Aronson, Elliot, and Carlsmith, J. Merill. 1968. “Experimentation in Social Psychology.” In Handbook of Social Psychology. 2nd ed., eds. Lindzey, Gardner and Aronson, Elliot. Reading, MA: Addison-Wesley, 1–79.Google Scholar
Aronson, Elliot, Wilson, Timothy D., and Brewer, Marilynn B.. 1998. “Experimentation in Social Psychology.” In The Handbook of Social Psychology. 4th ed., eds. Gilbert, Daniel T., Fiske, Susan T., and Lindzey, Gardner. Boston: McGraw-Hill, 99–142.Google Scholar
Barabas, Jason, and Jerit, Jennifer. 2010. “Are Survey Experiments Externally Valid?” American Political Science Review 104: 226–42.CrossRefGoogle Scholar
Baron, Reuben M., and Kenny, David A.. 1986. “The Moderator–Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations.” Journal of Personality and Social Psychology 51: 1173–82.CrossRefGoogle ScholarPubMed
Benz, Matthias, and Meier, Stephan. 2008. “Do People Behave in Experiments as in the Field?: Evidence from Donations.” Experimental Economics 11: 268–81.CrossRefGoogle Scholar
Berkowitz, Leonard, and Donnerstein, Edward. 1982. “External Validity Is More Than Skin Deep: Some Answers to Criticisms of Laboratory Experiments.” American Psychologist 37: 245–57.CrossRefGoogle Scholar
Campbell, Angus, Converse, Philip, Miller, Warren, and Stokes, Donald. 1960. The American Voter. Chicago: The University of Chicago Press. [Reprinted in 1980.]Google Scholar
Campbell, Donald T. 1969. “Prospective: Artifact and Control.” In Artifact in Behavioral Research, eds. Rosenthal, Robert and Rosnow, Robert. New York: Academic Press, 351–82.Google Scholar
Chong, Dennis, and Druckman, James N.. 2007a. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101: 637–55.CrossRefGoogle Scholar
Chong, Dennis, and Druckman, James N.. 2007b. “Framing Theory.” Annual Review of Political Science 10: 103–26.CrossRefGoogle Scholar
Converse, Jean M., and Schuman, Howard. 1974. Conversations at Random: Survey Research as Interviewers See It. New York: Wiley.Google Scholar
Depositario, Dinah Pura T., Nayga, Rodolfo M.., Wu, Ximing, and Laude, Tiffany P.. 2009. “Should Students Be Used as Subjects in Experimental Auctions?” Economic Letters 102: 122–24.CrossRefGoogle Scholar
Dickhaut, John W., Leslie Livingstone, J., and Watson, David J. H.. 1972. “On the Use of Surrogates in Behavioral Experimentation.” The Accounting Review 47 (Suppl): 455–71.Google Scholar
Druckman, James N. 2001. “On the Limits of Framing Effects.” Journal of Politics 63: 1041–66.CrossRefGoogle Scholar
Druckman, James N. 2009. “Competing Frames in a Campaign.” Unpublished paper, Northwestern University.Google Scholar
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. “The Growth and Development of Experimental Research Political Science.” American Political Science Review 100: 627–36.CrossRefGoogle Scholar
Franklin, Charles H. 1991. “Efficient Estimation in Experiments.” The Political Methodologist 4: 13–15.Google Scholar
Friedman, Daniel, and Sunder, Shyam. 1994. Experimental Economics: A Primer for Economists. New York: Cambridge University Press.CrossRefGoogle Scholar
Friedman, Milton. 1953. Essays in Positive Economics. Chicago: The University of Chicago Press.Google Scholar
Gaines, Brian J., Kuklinski, James H., and Quirk, Paul J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15: 1–20.CrossRefGoogle Scholar
Gerber, Alan S., Gimpel, James G., Green, Donald P., and Shaw, Daron R.. 2007. “The Influence of Television and Radio Advertising on Candidate Evaluations: Results from a Large Scale Randomized Experiment.” Unpublished paper, Yale University.
Gerber, Alan S., and Green, Donald P.. 2008. “Field Experiments and Natural Experiments.” In The Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David. Oxford: Oxford University Press, 357–81.Google Scholar
Gordon, Michael E., Allen Slade, L., and Schmitt, Neal. 1986. “The ‘Science of the Sophomore’ Revisited: From Conjecture to Empiricism.” Academy of Management Review 11: 191–207.Google Scholar
Groves, Robert M., and Peytcheva, Emilia. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72: 167–89.CrossRefGoogle Scholar
Guala, Francesco. 2005. The Methodology of Experimental Economics. New York: Cambridge University Press.CrossRefGoogle Scholar
Henrich, Joseph, Heine, Steven J., and Norenzayan, Ara. 2009. “The Weirdest People in the World: How Representative Are Experimental Findings from American University Students? What Do We Really Know about Human Psychology?” Unpublished paper, University of British Columbia.
James, William L., and Sonner, Brenda S.. 2001. “Just Say No to Traditional Student Samples.” Journal of Advertising Research 41: 63–71.Google Scholar
Kam, Cindy D. 2005. “Who Toes the Party Line?: Cues, Values, and Individual Differences.” Political Behavior 27: 163–82.CrossRefGoogle Scholar
Kam, Cindy D. 2007. “When Duty Calls, Do Citizens Answer?” Journal of Politics 69: 17–29.CrossRefGoogle Scholar
Kam, Cindy D., and Franzese, Jr Robert J.. 2007. Modeling and Interpreting Interactive Hypotheses in Regression Analysis. Ann Arbor: University of Michigan Press.Google Scholar
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29: 415–40.CrossRefGoogle Scholar
Kinder, Donald R. 1998. “Communication and Opinion.” Annual Review of Political Science 1: 167–97.CrossRefGoogle Scholar
Kruglanski, Arie W. 1975. “The Human Subject in the Psychology Experiment: Fact and Artifact.” In Advances in Experimental Social Psychology, ed. Berkowitz, Leonard. New York: Academic Press, 101–47.Google Scholar
Lau, Richard R., and Redlawsk, David P.. 2006. How Voters Decide: Information Processing in Election Campaigns. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Liyanarachchi, Gregory A. 2007. “Feasibility of Using Student Subjects in Accounting Experiments: A Review.” Pacific Accounting Review 19: 47–67.CrossRefGoogle Scholar
MacDonald, Paul. 2003. “Useful Fiction or Miracle Maker: The Competing Epistemological Foundations of Rational Choice Theory.” American Political Science Review 97: 551–65.CrossRefGoogle Scholar
McDermott, Rose. 2002. “Experimental Methodology in Political Science.” Political Analysis 10: 325–42.CrossRefGoogle Scholar
Milgram, Stanley. 1963. “Behavioral Study of Obedience.” Journal of Abnormal and Social Psychology 67: 371–78.CrossRefGoogle ScholarPubMed
Mintz, Alex, Redd, Steven B., and Vedlitz, Arnold. 2006. “Can We Generalize from Student Experiments to the Real World in Political Science, Military Affairs, and International Relations?” Journal of Conflict Resolution 50: 757–76.CrossRefGoogle Scholar
Mook, Douglas G. 1983. “In Defense of External Invalidity.” American Psychologist 38: 379–87.CrossRefGoogle Scholar
Morton, Rebecca B., and Williams, Kenneth C.. 2008. “Experimentation in Political Science.” In The Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David. Oxford: Oxford University Press, 339–56.Google Scholar
Peterson, Robert A. 2001. “On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-Analysis.” Journal of Consumer Research 28: 450–61.CrossRefGoogle Scholar
Plott, Charles R. 1991. “Will Economics Become an Experimental Science?” Southern Economic Journal 57: 901–19.CrossRefGoogle Scholar
Roth, Alvin E. 1995. “Introduction to Experimental Economics.” In The Handbook of Experimental Economics, eds. Kagel, John H. and Roth, Alvin E.. Princeton, NJ: Princeton University Press, 3–109.Google Scholar
Sears, David O. 1986. “College Sophomores in the Laboratory: Influence of a Narrow Data Base on Social Psychology's View of Human Nature.” Journal of Personality and Social Psychology 51: 515–30.CrossRefGoogle Scholar
Shadish, William R., Cook, Thomas D., and Campbell, Donald T.. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.Google Scholar
Simon, Herbert A. 1963. “Problems of Methodology Discussion.” American Economic Review Proceedings 53: 229–31.Google Scholar
Simon, Herbert A. 1979. “Rational Decision Making in Business Organizations.” American Economic Review 69: 493–513.Google Scholar
Slothuus, Rune. 2009. “The Political Logic of Party Cues in Opinion Formation.” Paper presented at the annual meeting of the Midwest Political Science Association, Chicago.
Sniderman, Paul M., Brody, Richard A., and Tetlock, Philip E.. 1991. Reasoning and Choice: Explorations in Political Psychology. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Sniderman, Paul M., and Theriault, Sean M.. 2004. “The Structure of Political Argument and the Logic of Issue Framing.” In Studies in Public Opinion, eds. Saris, Willem E. and Sniderman, Paul M.. Princeton, NJ: Princeton University Press, 133–65.Google Scholar
Stevens, Charles D., and Ash, Ronald A.. 2001. “The Conscientiousness of Students in Subject Pools: Implications of ‘Laboratory’ Research.” Journal of Research in Personality 35: 91–97.CrossRefGoogle Scholar
Transue, John E., Lee, Daniel J., and Aldrich, John H.. 2009. “Treatment Spillover Effects across Survey Experiments.” Political Analysis 17: 143–61.CrossRefGoogle Scholar
Zaller, John. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge University Press.CrossRefGoogle Scholar
Zimbardo, Phillip. 1973. “A Pirandellian Prison.” New York Times Magazine, April 8.

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×