Top

TechTrends

Published in:

16-07-2022 | Original Paper

Lessons Learned from Online Learning at Scale: a Study of Exemplar Learning Organizations

Authors: Siyuan Li, Scotty D. Craig, Noah L. Schroeder

Published in: TechTrends | Issue 1/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

As the areas of education and workforce training tend more toward online training, they have emphasized the rapid setup and expansion of online environments. This scaling-up and scaling-out of training programs can be difficult, but there are examples of success available for us to learn from. To capture the lessons learned, we conducted semi-structured interviews across academic, public, private, and non-profit organizations. Using deductive thematic analysis, three major groups of determinants were identified: technological infrastructure, human infrastructure, and governance. A framework consisting of three actionable recommendations was created to facilitate online learning organizational scaling-up including learner (learning experiences and user-friendly design), technical (technical competencies and the use of integrated data), and management (technical competencies for both learners and instructors as well as the use of integrated data) factors.

previous article Instructional Design for Socially Distanced Compliance Audit

next article A Categorization of Workplace Learning Goals for Multi-Stakeholder Recommender Systems: A Systematic Review

Available only for authorised users

Abouchedid, K., & Eid, G. M. (2004). E-learning challenges in the Arab World: revelations from a case study profile. Quality Assurance in Educational, 12(1), 15–27. https://doi.org/10.1108/09684880410517405CrossRef

Alrasheedi, M., & Capretz, L. (2015). Determination of Critical Success Factors Affecting Mobile Learning: A Meta-Analysis Approach. Turkish Online Journal of Educational Technology, 14(2), 41–51. Retrieved January 15, 2021, from https://arxiv.org/ftp/arxiv/papers/1801/1801.04288.pdf

Bari, M., Djouab, R., & Hoa, C. P. (2018). Elearning current situation and emerging challenges. PEOPLE: International Journal of Social Sciences, 4(2), https://doi.org/10.20319/pijss.2018.42.97109

Bird, J., & Morgan, C. (2003). Adults contemplating university study at a distance: Issues, themes and concerns. The International Review of Research in Open and Distributed Learning, 4(1). https://doi.org/10.19173/irrodl.v4i1.130

Burnette, D. M. (2015). Negotiating the mine field: Strategies for effective online education administrative leadership in higher education institutions. Quarterly Review of Distance Education, 16(3), 13–25.

Carrillo, C., & Flores, M. A. (2020). Covid-19 and teacher education: A literature review of online teaching and learning practices. European Journal of Teacher Education, 43(4), 466–487. https://doi.org/10.1080/02619768.2020.1821184CrossRef

Cheok, M. L., & Wong, S. L. (2015). Predictors of e-learning satisfaction in teaching and learning for school teachers: A literature review. International Journal of Instruction, 8(1), 75–90. Retrieved January 15, 2021, https://eric.ed.gov/?id=EJ1085289

Chitura, M. (2015). Making it happen with distance education: Critical concerns for student support services in open and distance learning. Greener Journal of Educational Research, 5(1), 009–016. https://doi.org/10.15580/GJER.2015.1.012315019CrossRef

Clarke, J., & Dede, C. (2009). Design for scalability: A case study of the River City curriculum. Journal of Science Education and Technology, 18(4), 353–365.CrossRef

Coburn, C. E. (2003). Rethinking scale: moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3–12. https://doi.org/10.3102/0013189X032006003CrossRef

Collins, C., Buhalis, D., & Peters, M. (2003). Enhancing SMEs business performance through the internet and E-learning platforms. Education + Training, 45(8), 483–494. https://doi.org/10.1108/00400910310508874CrossRef

Cook, V., & Uranis, J. (2019). Supporting online program quality through online enterprise-level standards. In Ensuring Quality and Integrity in Online Learning Programs (pp. 254–280). IGI Global. Retrieved January 15, 2021, from https://www.igi-global.com/chapter/supporting-online-program-quality-through-online-enterprise-level-standards/225268

Coursetune. (n.d.). Retrieved May 9 2022 from https://coursetune.com/

Craig, S. D., Hu, X., Graesser, A. C., Bargagliotti, A. E., Sterbinsky, A., Cheney, K. R., & Okwumabua, T. (2013). The impact of a technology-based mathematics after-school program using ALEKS on student's knowledge and behaviors. Computers & Education, 68, 495–504. https://doi.org/10.1016/j.compedu.2013.06.010

Craig, S. D., & Douglas, I. (2019). Distributed learning instructional theories. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 43–60). Washington, DC: Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/

Craig, S. D., & Schroeder, N. L. (2020). Science of learning and readiness (SoLaR) recommendation report: science of learning practices for distributed online environments. Arizona State University Tempe United States. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/pdfs/AD1105006.pdf

Craig, S. D., Schroeder, N. L., Roscoe, R. D., Cooke, N. J., Prewitt, D., Siyuan, L., ... & Clark, A. (2020a). Science of learning and readiness state of the art report. Arizona State University Tempe United States. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/pdfs/AD1106815.pdf

Craig, S. D., Li, S., Prewitt, D., Morgan, L. A., & Schroeder, N. L. (2020b). Science of learning and readiness (SoLaR) exemplar report: A path toward learning at scale. Arizona State University. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/citations/AD1104999.pdf

Craig, S. D., Siegel, R. F., & Li, S. (2021). An application of the learning science evaluation checklist: use case report. Arizona State University. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/citations/AD1141190.pdf

Crompton, H., & Burke, D. (2018). The use of mobile learning in higher education: A systematic review. Computers & Education, 123, 53–64. Retrieved May 12, 2022 from https://www.learntechlib.org/p/201270/

Davis, D., Chen, G., Hauff, C., & Houben, G. J. (2018). Activating learning at scale: A review of innovations in online learning strategies. Computers & Education, 125, 327–344. https://doi.org/10.1016/j.compedu.2018.05.019CrossRef

Dede, C. (2005). Scaling up: evolving innovations beyond ideal settings to challenging contexts of practice. In R. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 551–566). Cambridge University Press. https://doi.org/10.1017/CBO9780511816833.034

Dede, C., Richards, J., & Saxberg, B. (Eds.). (2019). Learning engineering for online education: Theoretical contexts and design-based examples. Routledge. https://doi.org/10.37074/jalt.2019.2.2.17

DeMonte, J. (2013). High-quality professional development for teachers: Supporting teacher training to improve student learning. Center for American Progress. Retrieved May 12, 2022 from https://cdn.americanprogress.org/wp-content/uploads/2013/07/DeMonteLearning4Teachers-1.pdf

Denzin, N. K. (2017). The sociological interview. The Research Act, 122–143. https://doi.org/10.4324/9781315134543-6

D’Mello, S. K., Picard, R., & Graesser, A. C. (2007). Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems, 22(4), 53–61. https://doi.org/10.1109/MIS.2007.79CrossRef

Dooley, K. E., & Murphrey, T. P. (2000). How the perspectives of administrators, faculty, and support units impact the rate of distance education adoption. Online Journal of Distance Learning Administration, 3(4), 1–12. Retrieved May 12, 2022 from https://www.learntechlib.org/p/92503/

Elmore, R. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–27. https://doi.org/10.17763/haer.66.1.g73266758j348t33CrossRef

Erb, S., & Shaw, R. (2019). Culture Change. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 339–356). Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/

Farid, S., Qadir, M., Ahmed, M. U., & Khattak, M. D. (2018). Critical success factors of E-learning systems: a quality perspective. Pakistan Journal of Distance and Online Learning, 4(1), 1–20. Retrieved January 15, 2021, from https://files.eric.ed.gov/fulltext/EJ1267239.pdf

Finelli, C. J., Ott, M., Gottfried, A. C., Hershock, C., O’neal, C., & Kaplan, M. (2008). Utilizing instructional consultations to enhance the teaching performance of engineering faculty. Journal of Engineering Education, 97(4), 397–411. https://doi.org/10.1002/j.2168-9830.2008.tb00989.xCrossRef

Fishman, B., Dede, C., & Means, B. (2016). Teaching and technology: New tools for new times. Handbook of research on teaching, 1269–1334. https://doi.org/10.3102/978-0-935302-48-6_21

Floyd, D. L., & Casey-Powell, D. (2004). New roles for student support services in distance learning. New Directions for Community Colleges, 128, 55–64. Retrieved May 12, 2022 from https://www.learntechlib.org/p/100773/

Giattino, T., & Stafford, M. (2019). Governance for learning ecosystems. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 317–338). Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/

Hafeez, A., Gujjar, A. A., & Noreen, Z. (2014). Demanding need of growing technologies in distance learning system. Turkish Online Journal of Distance Education, 15(4), 170–180. https://doi.org/10.17718/tojde.04872CrossRef

Hai-Jew, S. (2006). Operationalizing trust: Building the online trust student survey (OTSS). Journal of Interactive Instruction Development, 19(2), 16–30. Retrieved January 15, 2021, from http://www.salt.org/salt.asp?ss=l&pn=jiid

Herzberg, F. I. (1966). Work and the nature of man. World.

Hong, S., Thong, J. Y., & Tam, K. Y. (2006). Understanding continued information technology usage behavior: A comparison of three models in the context of mobile internet. Decision Support Systems, 42(3), 1819–1834. https://doi.org/10.1016/j.dss.2006.03.009CrossRef

Hunt, T., Carter, R., Zhang, L., & Yang, S. (2020). Micro-credentials: The potential of personalized professional development. Development and Learning in Organizations: An International Journal. Retrieved January 15, 2021, from https://www.tcd.ie/media/tcd/content-assets/pdf/Micro-credentials-The-potential-of-personalized-professional-development.pdf

iDesign. (n.d.). Retrieved May 9 2022 from https://www.idesignedu.org/

Kaplan, A. M., & Haenlein, M. (2016). Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the Cookie Monster. Business horizons, 59(4), 441–450. https://doi.org/10.1016/j.bushor.2016.03.008CrossRef

Keengwe, J. (Ed.). (2018). Handbook of research on virtual training and mentoring of online instructors. IGI Global.

Khalil, H., & Ebner, M. (2014). MOOCs Completion Rates and Possible Methods to Improve Retention - A Literature Review. In J. Viteli & M. Leikomaa (Eds.), Proceedings of EdMedia 2014–World Conference on Educational Media and Technology (pp. 1305–1313). Association for the Advancement of Computing in Education (AACE). Retrieved December 8, 2020 from https://www.learntechlib.org/primary/p/147656/

Kim, S. W. (2016). MOOCs in higher education. Virtual learning, 121–135. https://doi.org/10.5772/66137

Lancaster, J. W., Stein, S. M., MacLean, L. G., Van Amburgh, J., & Persky, A. M. (2014). Faculty development program models to advance teaching and learning within health science programs. American Journal of Pharmaceutical Education, 78(5). https://www.ajpe.org/content/78/5/99

Lee, J. W. (2010). Online support service quality, online learning acceptance, and student satisfaction. Internet and Higher Education, 13, 277–283. https://doi.org/10.1016/j.iheduc.2010.08.002CrossRef

Li, S., Siegle, R. F., & Craig, S. D. (2021). Case report for the learning science evaluation checklist. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 65, No. 1, pp. 561–565). Sage CA: Los Angeles, CA: SAGE Publications.

Lisewski, B. (2004). Implementing a learning technology strategy: Top-down strategy meets bottom-up culture. Association for Learning Technology: Research in Learning Technology, 12(2), 175–188. https://doi.org/10.1080/0968776042000216228CrossRef

Liu, M., Kang, J., & McKelroy, E. (2015). Examining learners’ perspective of taking a MOOC: reasons, excitement, and perception of usefulness. Educational Media International, 52(2), 129–146. https://doi.org/10.1080/09523987.2015.1053289CrossRef

Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008–2012. International Review of Research in Open and Distributed Learning, 14(3), 202–227 Retrieved January 15, 2021, from https://centaur.reading.ac.uk/33109/

Liyanagunawardena, T. R., Scalzavara, S., & Williams, S. A. (2017). Open badges: a systematic review of peer-reviewed published literature (2011-2015). European Journal of Open, Distance and E-learning, 20(2), 1–16.

Luan, W. S., & Teo, T. (2009). Investigating the technology acceptance among student teachers in Malaysia: An application of the technology acceptance model (TAM). Asia-Pacific Education Researcher, 18(2), 1–1 Retrieved January 15, 2021, from. https://pdfs.semanticscholar.org/3b38/89162a95de50a60737b0f0087c5e2866253b.pdf?_ga=2.40160161.286461491.1611557988-176642363.1610760777

Ludwig-Hardman, S., & Dunlap, J. C. (2003). Learner support services for online students: Scaffolding for success. International Journal of Research in Open and Distance Learning, 4(1). https://doi.org/10.19173/irrodl.v4i1.131

Martin, S. M., & Trang, M. L. (2018). Creating personalized learning using aggregated data from students’ online conversational interactions in customized groups. Learning Engineering for Online Education: Theoretical Contexts and Design-Based Examples, 144–165.

McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. University of Prince Edward Island. Retrieved January 15, 2021, from https://www.oerknowledgecloud.org/archive/MOOC_Final.pdf

Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 1–47.CrossRef

Miller, S., Meier, E., Payne-Bourcy, L., Shablack, S., Newman, D. L., Wan, T. Y., & Pack, G. (2003). Technology use as a catalyst for change: A leadership framework for transforming urban teacher preparation. IEJLL: International Electronic Journal for Leadership in Learning, 7(12), 2461–2480 Retrieved January 15, 2021, from https://www.researchgate.net/publication/295922106_Technology_use_as_a_catalyst_for_change_A_leadership_framework_for_transforming_urban_teacher_preparation

Morel, R. P., Coburn, C., Catterson, A. K., & Higgs, J. (2019). The multiple meanings of scale: implications for researchers and practitioners. Educational Researcher, 48(6), 369–377. https://doi.org/10.3102/0013189X19860531CrossRef

Motiwalla, L. F. (2007). Mobile learning: A framework and evaluation. Computers & Education, 49(3), 581–596. https://doi.org/10.1016/j.compedu.2005.10.011CrossRef

Mulder, F., & Jansen, D. (2015). MOOCs for opening up education and the OpenupEd initiative. In C. J. Bonk, M. M. Lee, T. C. Reeves, & T. H. Reynolds (Eds.), MOOCs and open education around the world. Routledge. https://doi.org/10.4324/9781315751108-16

Newby, T., & Cheng, Z. (2019). Instructional design badges. Education Technology Research Development, 68, 1053–1067. https://doi.org/10.1007/s11423-019-09719-7CrossRef

Nichols, M. (2010). Student perceptions of support services and the influence of targeted interventions on retention in distance education. Distance education, 31(1), 93–113. https://doi.org/10.1080/01587911003725048CrossRef

Obexer, R. (2018). Scaling online learning: the case for a programme-level approach. In A. Altmann, B. Ebersberger, C. Mössenlechner, & D. Wieser (Eds.), The Disruptive Power of Online Education (pp. 7–25). Emerald Publishing Limited. https://doi.org/10.1108/978-1-78754-325-620181002

Park, J. H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Journal of Educational Technology & Society, 12(4), 207–217.

Patru, M., & Balaji, V. (2016). Making Sense of MOOCs: A Guide for Policy-Makers in Developing Countries. http://oasis.col.org/bitstream/handle/11599/2356/2016_Guide-on-MOOCs-for-Policy-Makers-in-Developing-Countries.pdf?sequence=4&isAllowed=y

Pelletier, K., Brown, M., Brooks, D. C., McCormack, M., Reeves, J., Arbino, N. … Mondelli, V. (2021). 2021 EDUCAUSE Horizon Report Teaching and Learning Edition. EDU. Retrieved May 12, 2022 from https://www.learntechlib.org/p/219489/

Rawls, T. (2016, February 8). Achieving economies of scale in Higher Education. The EvoLLLution. Retrieved April 11, 2022, from https://evolllution.com/managing-institution/higher_ed_business/achieving-economies-of-scale-in-higher-education/

Rees-Johnstone, E. (2016, February 19). Four levers for scaling higher education. The EvoLLLution. Computers & Education, 68, 495–504. Retrieved April 11, 2022, from https://evolllution.com/managing-institution/higher_ed_business/four-levers-for-scaling-higher-education/dbehaviors

Roll, I., Russell, D. M., & Gašević, D. (2018). Learning at scale. International Journal of Artificial Intelligence in Education, 28(4), 471–477. https://doi.org/10.1007/s40593-018-0170-7CrossRef

Rovai, A. P., & Downey, J. R. (2010). Why some distance education programs fail while others succeed in a global environment. Internet and Higher Education, 13(3), 141–147. Elsevier Ltd. Retrieved May 12, 2022 from https://www.learntechlib.org/p/108384/

Screnci, D. (2017, April 24). Building communities of practice to support EdTech. EDUCAUSE Review. Retrieved May 9, 2022, from https://er.educause.edu/articles/2017/4/building-communities-of-practice-to-support-edtech

Selvaratnam, R. M., & Sankey, M. D. (2021). An integrative literature review of the implementation of microcredentials in higher education: Implications for practice in Australasia. Journal of Teaching and Learning for Graduate Employability, 12(1), 1–17. https://doi.org/10.21153/JTLGE2021VOL12NO1ART942CrossRef

Smith, G. G., & Kurthen, H. (2007). Front-Stage and Back-Stage in Hybrid E-Learning Face-to-Face Courses. International Journal on E-Learning, 6(3), 455–474. Association for the Advancement of Computing in Education (AACE). Retrieved May 11, 2022 from https://www.learntechlib.org/primary/p/20008/

Sohoni, S., Craig, S. D., & Vedula, K. (2017). A blueprint for an ecosystem for supporting high quality education for engineering. Journal of Engineering Education Transformations, 30(4), 58–66. https://doi.org/10.16920/jeet/2017/v30i4/114107

Technology for Education - Eberly Center - Carnegie Mellon University. (n.d.). Retrieved May 9 2022, from https://www.cmu.edu/teaching/technology/index.html

Wang, X., Talluri, S. T., & Rose, C., & Koedinger, K. (2019). UpGrade: sourcing student open-ended solutions to create scalable learning opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale. New York, NY, USA, Article 17, 1–10. https://doi.org/10.1145/3330430.3333614

Westley, F., Antadze, N., Riddell, D. J., Robinson, K., & Geobey, S. (2014). Five configurations for scaling up social innovation: case examples of nonprofit organizations from Canada. The Journal of Applied Behavioral Science, 50(3), 234–260. https://doi.org/10.1177/0021886314532945CrossRef

Wong, D. (2007). A critical literature review on e-learning limitations. Journal for the Advancement of Science and Arts, 2(1), 55–62 Retrieved January 15, 2021, from, https://www.researchgate.net/profile/Dominic_Wong3/publication/228677287_A_Critical_Literature_Review_on_e-Learning_Limitations/links/57a0a26a08ae5f8b25891cee.pdf

Wylie, C., & MacDonald, J. (2020). Scaling up research-based educational practices. Oxford University Press. https://doi.org/10.1093/OBO/9780199756810-0254

Title: Lessons Learned from Online Learning at Scale: a Study of Exemplar Learning Organizations
Authors: Siyuan Li
Scotty D. Craig
Noah L. Schroeder
Publication date: 16-07-2022
Publisher: Springer US
Published in: TechTrends / Issue 1/2023
Print ISSN: 8756-3894
Electronic ISSN: 1559-7075
DOI: https://doi.org/10.1007/s11528-022-00761-6

Mobile Performance Support Systems: Characteristics, Benefits, and Conditions

Original Paper

Deep Learning Becomes a Reality when Emerging Technologies Meet Learning Design

Original Paper

The History of Robotics and Implications for K-12 STEM Education

Column: History Corner

Organizational Analysis in Preparation for LMS Change: a Narrative Case Study

Original Paper

A Decade of Rethinking Creativity, Technology and Learning: Reflections with the Deep-Play Research Group

Column: Rethinking Creativity and Technology in Education

K12 Practitioners’ Experiences Implementing Human Performance Improvement: Processes, Products, and Potentials

Original Paper

Springer Professional

Lessons Learned from Online Learning at Scale: a Study of Exemplar Learning Organizations

Abstract

Other articles of this Issue 1/2023

Mobile Performance Support Systems: Characteristics, Benefits, and Conditions

Deep Learning Becomes a Reality when Emerging Technologies Meet Learning Design

The History of Robotics and Implications for K-12 STEM Education

Organizational Analysis in Preparation for LMS Change: a Narrative Case Study

A Decade of Rethinking Creativity, Technology and Learning: Reflections with the Deep-Play Research Group

K12 Practitioners’ Experiences Implementing Human Performance Improvement: Processes, Products, and Potentials

Premium Partner

Springer Professional

Abstract

Please log in to get access to your license.

Mobile Performance Support Systems: Characteristics, Benefits, and Conditions

Deep Learning Becomes a Reality when Emerging Technologies Meet Learning Design

The History of Robotics and Implications for K-12 STEM Education

Organizational Analysis in Preparation for LMS Change: a Narrative Case Study

A Decade of Rethinking Creativity, Technology and Learning: Reflections with the Deep-Play Research Group

K12 Practitioners’ Experiences Implementing Human Performance Improvement: Processes, Products, and Potentials