Skip to main content

Advertisement

SpringerLink
Log in

Assessing the Effectiveness of a College Freshman Seminar Using Propensity Score Adjustments

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Researchers investigated the impact that a first-year college experience course had on students’ first-year grade point averages (GPAs) and retention rates. A sample of 109 first-year students enrolled in the course was compared to a sample of 326 students from the same university who had not taken the course. The goals of the experience course were to reduce attrition, increase grade point averages, and enhance academic skills. Without accounting for selection bias, those who took the course had similar retention rates and lower GPAs than those who did not take the course. After matching on propensity scores, the negative effects of the program on GPA were nullified and those in the program were more likely to enroll for a second year. Although the benefits from the course were weak, the positive impact of the program was more apparent after accounting for individual differences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • ACT. (2006). Retrieved March 12, 2009, from National Collegiate Retention and Persistence to Degree Rates Web site: http://www.act.org/research/policymakers/pdf/retain_2006.pdf.

  • Aitken, N. D. (1982). College student performance, satisfaction and retention: Specification and estimation of a structural model. The Journal of Higher Education, 53, 32–50.

    Article  Google Scholar 

  • Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23–35.

    Article  Google Scholar 

  • Allison, P. (2001). Missing data. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Astin, A. W. (1993). What matters most in college? Four critical years revisited. San Francisco: Jossey-Bass.

    Google Scholar 

  • Austin, P. C., Grootendorst, P., & Anderson, G. M. (2007). A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine, 26, 734–753.

    Article  Google Scholar 

  • Barefoot, B. O. (2000). The first-year experience: Are we making it any better? About Campus, 4(6), 12–18.

    Google Scholar 

  • Barefoot, B. O., & Fidler, P. P. (1992). The 1991 national survey of freshman seminar programming (Monograph no. 10). Columbia, SC: National Resource Center for the Freshman Year Experience, University South Carolina.

    Google Scholar 

  • Beck, A. T., & Beck, R. W. (1972). Screening depressed patients in family practice: A rapid technic. Postgraduate Medicine, 52, 81–85.

    Google Scholar 

  • Burgette, J., & Magun-Jackson, S. (2009). Freshman orientation, persistence, and achievement: A longitudinal analysis. Journal of College Retention: Research, Theory and Practice, 10(3), 235–263.

    Article  Google Scholar 

  • Busato, V. V., Prins, F. J., Elshout, J. J., & Hamaker, C. (2000). Intellectual ability, learning style, personality, achievement motivation and academic success among psychology students in higher education. Personality and Individual Differences, 29, 1057–1068.

    Article  Google Scholar 

  • Cabrera, A. F., Nora, A., & Castaneda, M. B. (1993). College persistence: Structural equations modeling test of an integrated model of student retention. The Journal of Higher Education, 64, 123–139.

    Article  Google Scholar 

  • Cepeda, M. S., Boston, R., Farrar, J. T., & Strom, B. L. (2003). Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. American Journal of Epidemiology, 158, 280–287.

    Article  Google Scholar 

  • Cochran, W. G., & Rubin, D. B. (1973). Controlling bias in observational studies: A review. Sankhya, Series A, 35, 417–446.

    Google Scholar 

  • Cokley, K. O. (2003). What do we know about the motivation of African American students? Challenging the “anti-intellectual” myth. Harvard Educational Review, 73, 524–558.

    Google Scholar 

  • DeBerard, M. S., Spielmans, G. I., & Julka, D. C. (2004). Predictors of academic achievement and retention among college freshman: A longitudinal study. College Student Journal, 38, 66–80.

    Google Scholar 

  • Engberg, M. E., & Mayhew, M. J. (2007). The influence of first-year “success” courses on student learning and democratic outcomes. Journal of College Student Development, 48, 241–258.

    Article  Google Scholar 

  • Fox, R. N. (1984). Reliability and discriminant validity of institutional integration scales for disadvantaged college students. Educational and Psychological Measurement, 44, 1051–1057.

    Article  Google Scholar 

  • Friedman, D. B., & Alexander, J. S. (2007). Investigating a first-year seminar as an anchor course in learning communities. Journal of the First-Year Experience & Students in Transition, 19, 63–74.

    Google Scholar 

  • Gibson, C. M. (2003). Privileging the participant: The importance of sub-group analysis in social welfare evaluations. American Journal of Evaluation, 24, 443–469.

    Google Scholar 

  • Gifford, D. D., Briceno-Perriott, J., & Mianzo, F. (2006). Locus of control: Academic achievement and retention in a sample of university first-year students. Journal of College Admission, 191, 18–25.

    Google Scholar 

  • Ginter, E. J., & Dwinell, P. L. (1994). The importance of perceived duration: Loneliness and its relationship to self-esteem and academic performance. Journal of College Student Development, 35, 456–460.

    Google Scholar 

  • Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg: Tilburg University Press.

    Google Scholar 

  • Goldberg, L. R. (2004). Big-five factor markers derived from the IPIP item pool (50 items). International personality item pool: A scientific collaboratory for the development of advanced measures of personality and other individual differences. Retrieved July 17, 2004, from http://ipip.ori.org/newQform50b5.htm.

  • Gow, A. J., Whiteman, M. C., Pattie, A., & Deary, I. J. (2005). Goldberg’s ‘IPIP’ Big-Five factor markers: Internal consistency and concurrent validation in Scotland. Personality and Individual Difference, 39, 317–329.

    Article  Google Scholar 

  • Grayson, J. P. (1988). Racial origin and student retention in a Canadian University. Higher Education, 36, 323–352.

    Article  Google Scholar 

  • Grunwald, H. E., & Mayhew, M. J. (2008). Using propensity scores for estimating causal effects: A study in the development of moral reasoning. Research in Higher Education, 49, 758–775.

    Article  Google Scholar 

  • Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2, 405–420.

    Article  Google Scholar 

  • Handel, D. D. (2001, April). The relative contribution of participating in a first-year seminar on student satisfaction and retention into the sophomore year. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. (ERIC Document Reproduction Service No. ED453724)

  • Harackiewicz, J. M., Barron, K. E., Tauer, J. M., & Elliot, A. J. (2002). Predicting success in college: A longitudinal study of achievement goals and ability measures as predictors of interest and performance from freshman year through graduation. Journal of Educational Psychology, 94, 562–575.

    Article  Google Scholar 

  • Hirano, K., & Imbens, G. W. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2, 259–278.

    Article  Google Scholar 

  • Hunter, M. S. (Summer, 2006). Fostering student learning and success through first-year programs. Association of American Colleges and Universities, 1–7.

  • Jamelske, E. (2009). Measuring the impact of a university first-year experience program on student GPA and retention. Higher Education, 57, 373–391.

    Article  Google Scholar 

  • Keup, J. R. (2005). The impact of curricular interventions on intended second year re-enrollment. Journal of College Student Retention: Research, Theory and Practice, 7, 61–89.

    Article  Google Scholar 

  • Komarraju, M., Karau, S., & Schmeck, R. (2009). Role of the Big Five personality traits in predicting college students’ academic motivation and achievement. Learning and Individual Differences, 19, 47–52.

    Article  Google Scholar 

  • Koutsoubakis, D. (1999). A test of the effectiveness of a one-term freshman orientation program at the foreign campus of an accredited private American university. Journal of the First-Year Experience, 11, 33–58.

    Google Scholar 

  • Krause, K. L., Hartley, R., James, R., & McInnis, C. (2005). The first year experience in Australian universities: Findings from a decade of national studies. Centre for the Study of Higher Education University of Melbourne, Government report.

  • Krum, T. E., Costley, L. E., & Mattanah, J. (2009). Longitudinally predicting attrition among first and second year college students. Unpublished manuscript, Towson University, Towson, MD.

  • Kurth, T., Walker, A. M., Glynn, R. J., Chan, K. A., Gaziano, J. M., Berger, K., et al. (2006). Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. American Journal of Epidemiology, 163, 262–270.

    Article  Google Scholar 

  • Lane, D. J., & Gibbons, F. X. (2007). Am I the typical student? Perceived similarity to student prototypes predicts success. Personality and Social Psychology Bulletin, 33, 1380–1391.

    Google Scholar 

  • Lang, D. J. (2007). The impact of a first-year experience course on the academic performance, persistence, and graduation rates of first-semester college students at a public research university. Journal of the First-Year Experience & Students in Transition, 19, 9–25.

    Google Scholar 

  • Leow, C., Marcus, S., Zanutto, E., & Boruch, R. (2004). Effects of advanced course-taking on math and science achievement: Addressing selection bias using propensity scores. American Journal of Evaluation, 25, 461–478.

    Google Scholar 

  • McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403–425.

    Article  Google Scholar 

  • McInnis, C. (2001). Researching the first year experience: Where to from here? Higher Education Research & Development, 20, 105–114.

    Google Scholar 

  • McKenzie, K., Gow, K., & Schweitzer, R. (2004). Exploring first-year academic achievement through structural equation modeling. Higher Education Research and Development, 23, 95–112.

    Article  Google Scholar 

  • McKenzie, K., & Schweitzer, R. (2001). Who succeeds at University? Factors predicting academic performance in first year Australian students. Higher Education Research and Development, 20, 21–33.

    Article  Google Scholar 

  • Miller, J. W., Janz, J. C., & Chen, C. (2007). The retention impact of a first-year seminar on students with varying pre-college academic performance. Journal of the First-Year Experience & Students in Transition, 19, 47–62.

    Google Scholar 

  • Napoli, A. R., & Wortman, P. M. (1998). Psychosocial factors related to retention and early departure of two-year community college students. Research in Higher Education, 39, 419–455.

    Article  Google Scholar 

  • National Resource Center for the First-Year Experience and Students in Transition and the Policy Center on the First Year of College. (2009). Retrieved March 20, 2009, from National Resource Center for the First-Year Experience and Students in Transition Web site: http://www.sc.edu/fye/index.html.

  • Nes, L. S., Evans, D. R., & Segerstrom, S. C. (2009). Optimism and college retention: Mediation by motivation, performance and adjustment. Journal of Applied Social Psychology, 39, 1887–1912.

    Article  Google Scholar 

  • Nicpon, M. F., Huser, L., Blanks, E. H., Sollenberger, S., Befort, C., & Robinson Kurpius, S. E. (2006). The relationship of loneliness and social support with college freshmen’s academic performance and persistence. Journal of College Student Retention: Research, Theory and Practice, 8, 345–358.

    Article  Google Scholar 

  • Noble, K., Flynn, N. T., Lee, J. D., & Hilton, D. (2007). Predicting successful college experiences: Evidence from a first year retention program. Journal of College Student Retention: Research, Theory and Practice, 9, 39–60.

    Article  Google Scholar 

  • Nora, A., Cabrera, A., Hagedorn, L., & Pascarella, E. (1996). Differential impacts of academic and social experiences on college-related behavioral outcomes across different ethnic and gender groups at four-year institutions. Research in Higher Education, 37, 427–451.

    Article  Google Scholar 

  • Norman, D. S. (2003). Loneliness and its effect of schooling and school achievement. Dissertation Abstracts International, 64(10), 3585. (UMI No. 3107462)

  • Pascarella, E. T., & Terenzini, P. T. (1980). Predicting freshman persistence and voluntary dropout decisions from a theoretical model. Journal of Higher Education, 51, 60–75.

    Article  Google Scholar 

  • Peterson, S. L. (1993). Career decision making self-efficacy and institutional integration of underprepared college students. Research in Higher Education, 34, 659–685.

    Article  Google Scholar 

  • Peterson, E. D., Pollack, C. V., Roe, M. T., Parsons, L. S., Littrell, K. A., Canto, J. G., et al. (2003). Early use of glycoprotein IIb/IIIa inhibitors in non-ST-elevation acute myocardial infarction: Observations from the National Registry of Myocardial Infarction 4. Journal of the American College of Cardiology, 2, 45–53.

    Article  Google Scholar 

  • Phillips, G. A. (1997). The utility and distinctiveness of the MMPI-2 College Maladjustment Scale. Unpublished master’s thesis, Western Carolina University.

  • Pickett, R. F. (2006). The importance of first-year experience courses. Unpublished manuscript, Southern Illinois University, Carbondale.

  • Porter, S. R., & Swing, R. L. (2006). Understanding how first-year seminars affect persistence. Research in Higher Education, 47, 89–109.

    Article  Google Scholar 

  • Reynolds, W. M., & Gould, J. W. (1981). A psychometric investigation of the standard and short form of the Beck Depression Inventory. Journal of Consulting and Clinical Psychology, 49, 306–307.

    Article  Google Scholar 

  • Rosenbaum, P. R. (1995). Observational studies. New York: Springer.

    Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

    Article  Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524.

    Article  Google Scholar 

  • Roweton, W. E. (1994). Predicting rural college retention among first-year undergraduates. (ERIC Document Reproduction Service No. ED370501)

  • Rubin, D. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2, 169–188.

    Article  Google Scholar 

  • Russell, D. W. (1996). UCLA Loneliness Scale (version 3): Reliability, validity, and factor structure. Journal of Personality Assessment, 66, 20–40.

    Article  Google Scholar 

  • Satterfield, C. D. (1999). Effects of institutional integration and career decision making self efficacy on academic persistence among college freshmen. Dissertation Abstracts International, 61, 886.

  • Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of American Statistical Association, 103, 1334–1344.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

    Google Scholar 

  • Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005). Propensity score methods gave similar results to traditional regression modeling in observational studies: A systematic review. Journal of Clinical Epidemiology, 58, 550–559.

    Article  Google Scholar 

  • Southern Illinois University. (2006a, August). University 101 course goal. Retrieved January 9, 2009, from Southern Illinois University’s University 101 Web site: http://www.siu.edu/~u101/goal.html.

  • Southern Illinois University. (2006b, August). University 101: Program overview. Retrieved January 9, 2009, from Southern Illinois University’s University 101 Web site: http://www.siu.edu/~u101/programoverview.html.

  • Southern Illinois University. (2008). Undergraduate catalog 2008–2009. Carbondale, IL: Southern Illinois University Carbondale.

    Google Scholar 

  • Strauss, L. C., & Volkwein, J. F. (2004). Predictors of student commitment at two-year and four-year institutions. The Journal of Higher Education, 75, 203–227.

    Article  Google Scholar 

  • Stürmer, T., Joshi, M., Glynn, R. J., Avorn, J., Rothman, K. J., & Schneeweiss, S. (2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437–447.

    Article  Google Scholar 

  • Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45, 89–125.

    Google Scholar 

  • Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: University of Chicago.

    Google Scholar 

  • Tinto, V. (1998). Colleges as communities: Taking research on student persistence seriously. The Review of Higher Education, 21, 167–177.

    Google Scholar 

  • Tobolowsky, B. F. (2006). Preliminary summary of results from the 2006 national survey on first-year seminars. Retrieved March 9, 2009, from National Resource Center for the First Year Experience & Students in Transition, University South Carolina Web site: http://www.sc.edu/fye/research/surveyfindings/surveys/survey06.html .

  • Tobolowsky, B. F., Mamrick, M., & Cox, B. E. (2005). The 2003 national survey of first-year seminars: Continuing innovations in the collegiate curriculum (Monograph no. 41). Columbia, SC: National Resource Center for the First Year Experience & Students in Transition, University South Carolina.

    Google Scholar 

  • Vallerand, R. J., & Bissonnette, R. (1992). Intrinsic, extrinsic, and amotivational styles as predictors of behavior: A prospective study. Journal of Personality, 60, 599–620.

    Article  Google Scholar 

  • Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C., & Vallières, E. F. (1992). The Academic Motivation Scale: A measure of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52, 1003–1017.

    Article  Google Scholar 

  • Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C. B., & Vallières, E. F. (1993). On the assessment of intrinsic, extrinsic, and amotivation in education: Evidence on the concurrent and construct validity of the Academic Motivation Scale. Educational and Psychological Measurement, 53, 159–172.

    Article  Google Scholar 

  • Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C. B., & Vallières, E. F. (2004). The Academic Motivation Scale (AMS-C 28) college version. Research Laboratory on Social Behavior. Retrieved http://www.er.uqam.ca/nobel/r26710/LRCS/echelles_en.htm.

  • Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004). Motivating learning, performance, and persistence: The synergistic effects of intrinsic goal contents and autonomy-supportive contexts. Journal of Personality and Social Psychology, 87, 246–260.

    Article  Google Scholar 

  • Viswesvaran, C., & Ones, D. S. (2000). Measurement error in “Big Five factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224–235.

    Article  Google Scholar 

  • Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educational and Psychological Measurement, 55, 177–185.

    Article  Google Scholar 

  • Wolf-Wendel, L. E., Tuttle, K., & Keller-Wolff, C. M. (1999). Assessment of a freshman summer transition program in an open-admissions institution. Journal of the First-Year Experience, 11, 7–32.

    Google Scholar 

  • Yockey, F. A., & George, A. A. (1998). The effects of a freshman seminar paired with supplemental instruction. Journal of the First-Year Experience, 10, 57–76.

    Google Scholar 

Download references

Acknowledgments

We would like to thank Virginia Rinella for providing information about the University 101 program; Michael E. Young and Sue M. Marcus for their statistical recommendations; and Matthew Herman, Vinetha Belur, Steven Middleton, Deborah Racey, Alen Avdic, and Blake Hutsell for collecting the data.

Author information

Authors and Affiliations

  1. Department of Educational and Human Sciences, College of Education, University of Central Florida, P.O. Box 161250, Orlando, FL, 32816-1250, USA

    M. H. Clark

  2. School of Management, University of Alaska Fairbanks, Fairbanks, AK, USA

    Nicole L. Cundiff

Authors
  1. M. H. Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicole L. Cundiff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. H. Clark.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Clark, M.H., Cundiff, N.L. Assessing the Effectiveness of a College Freshman Seminar Using Propensity Score Adjustments. Res High Educ 52, 616–639 (2011). https://doi.org/10.1007/s11162-010-9208-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-010-9208-x

Keywords

Search

Navigation