Introduction
Tracing the development of learning analytics… [it is possible to] highlight a gradual shift away from a technological focus towards an educational focus; the introduction of tools, initiatives and methods that are significant in the field today; and the issues that have not yet been addressed (Ferguson 2012, p. 305).
Learning analytics: key definitions and developments
Learning analytics: some historical notes
Challenges for learning analytics
Only a relatively low proportion of student success variation can be explained by traditional statistical modelling techniques such as multiple linear regression analyses. These techniques simply establish valid and reliable relationships between relatively few variables relevant to a specific context (Subotzky and Prinsloo 2011, p.183).
Methodology
Sample selection
Data analysis
Outcomes
Number of publications on LA in higher education by year, country of affiliation of first authors, journal, and language of publication for WoScc and SciELO databases
Country | Number of publications | % |
---|---|---|
USA | 78 | 20.3 |
Spain | 45 | 11.7 |
UK | 37 | 9.6 |
Australia | 31 | 8.1 |
China | 28 | 7.3 |
Taiwan | 22 | 5.7 |
Canada | 12 | 3.1 |
Netherlands | 11 | 2.9 |
Serbia, South Korea* | 9 | 2.3 |
Mexico | 8 | 2.1 |
Germany | 7 | 1.8 |
Japan, Norway, South Africaa | 6 | 1.6 |
Israel, Turkeya | 5 | 1.3 |
Greece, India, New Zealanda | 4 | 1.0 |
Belgium, Finland, Ireland, Pakistan, Saudi Arabia, Tunisiaa | 3 | 0.8 |
Austria, Chile, Colombia, Cyprus, Portugal, Romania, Sweden, united Arab Emiratesa | 2 | 0.5 |
Algeria, Bosnia and Herceg, Brazil, Costa Rica, Ecuador, Fiji, Iran, Italy, Lithuania, Malaysia, Montenegro, Peru, Switzerland, Thailanda | 1 | 0.3 |
Total | 385 | 100 |
Focus of the journal | Journals | Number of publications | % |
---|---|---|---|
Education and Technology | Computers & Education; British Journal of Educational Technology; Educational Technology & Society; Interactive Learning Environments; Internet and Higher Education; Etr&D-Educational Technology Research and Development; Journal of Computer Assisted Learning; Australasian Journal of Educational Technology; Journal of Computing In Higher Education; Acm Transactions on Computing Education; Computer Assisted Language Learning; Journal of Educational Computing Research; International Journal of Computer-Supported Collaborative Learning; Learning Media and Technology; International Review of Research in Open and Distance Learning; International Journal of Technology And Design Education; Journal Of Science Education and Technology; System; Recall | 174 | 45 |
Education | International Review of Research in Open And Distributed Learning; International; Journal of Educational Technology in Higher Education; Eurasia Journal of Mathematics Science and Technology Education; Educational Sciences-Theory & Practice; Distance Education; Physical Review Physics Education Research; Higher Education; Assessment & Evaluation in Higher Education; Teaching in Higher Education; Croatian Journal of Education-Hrvatski Casopis Za Odgoj I Obrazovanje; Research In Higher Education; Active Learning in Higher Education; Cadmo; Journal of Higher Education; Studies in Higher Education; Applied Measurement In Education; Studies In Educational Evaluation; Higher Education Policy; Journal of Educational Measurement; Thinking Skills and Creativity; British Educational Research Journal; Asia Pacific Education Review; Learning And Individual Differences; Cultura y Educacion; Journal of the Learning Sciences; Comunicar; Teachers College Record; Zeitschrift Fur Erziehungswissenschaft; Review of Higher Education; Innovations in Education and Teaching International; Ride; Revista Iberoamericana para la Investigación y el Desarrollo Educativo; Revista Electrónica de Investigación Educativa; Educación y Educadores; Estudios Pedagógicos; Innovación Educativa; Diálogos Sobre Educación. Temas Actuales en Investigación Educativa; Conrado; Revista Electrónica Educare; Apertura | 88 | 22,7 |
Education and Engineering | Ieee Transactions on Learning Technologies; Computer Applications in Engineering Education; International Journal of Engineering Education; Ieee Transactions on Education; International Journal of Electrical Engineering Education. | 87 | 22,5 |
Medical Education | Bmc Medical Education Medical Teacher; Academic Medicine; Advances in Physiology Education; Anatomical Sciences Education; European Journal of Dental Education; Teaching and Learning in Medicine; American Journal of Pharmaceutical Education; Journal of Continuing Education in the Health Professions; Medical Education; Journal of Surgical Education | 23 | 5,9 |
Other | Journal of Chemical Education; Journal of Educational Psychology; Cbe-Life Sciences Education; Journal of Geography in Higher Education; Biochemistry and Molecular; Biology Education; Assessing Writing; Language Learning; Physical Review Special Topics-Physics Education Research; Language Teaching; Educational Psychology; Journal of Hospitality Leisure Sport & Tourism Education | 13 | 3,4 |
Total | 385 | 100 |
Methodologies
Methodology | SciELO | WoScc | ||
---|---|---|---|---|
Number of publications | % | Number of publications | % | |
Quantitative | 6 | 60.0 | 293 | 78.1 |
Qualitative | 0 | 0.0 | 18 | 4.8 |
Mixed methods | 0 | 0.0 | 27 | 7.2 |
Non-empirical | 4 | 40.0 | 37 | 9.9 |
Total: | 10 | 100.0 | 375 | 100 |
Thematic analysis
Category | Description | Papers |
---|---|---|
1. Students | Learning design and the extent to which technologies may interfere in students’ learning (their autonomy, engagement, the ways in which they invest their time and effort and their progress in their learning) | |
2. Teachers | Lack of alignment between teachers’ pedagogical activities and LA. Also, the detachment between teachers and those responsible for LA (managers and administrators) | |
3. Educational theories | Lack of educational and pedagogical theories underpinning LA | |
4. Use of methods and data analysis | Use of highly technical mathematical models and quantitative techniques that include irrelevant attributes. Also, that the management of such large data sets is unduly time-consuming. Also, concerns about the ‘neutrality’ of data collection and techniques of analysis and the ways in which certain methods produce data which might affect results and have an impact on students’ learning. Finally, a concern about whether the methods actually measure learning | |
5. Research results | Diverse concerns about the results produced by LA. For example, the reduction of the complexities of learning into data; the lack of consideration of other learning factors or the broader context that cannot be measured; the loss of subjectivity and other factors involved in learning processes; the non-regulated cross-border use of data; and the ‘ecological validity’ of data | |
6. Data governance | Ways in which data are managed and used at micro (classroom), institutional and macro (national policies) levels so as to improve teaching and learning. Also, the lack of understanding about what to do with or how to use data. Also, a ‘managerialist’ approach to LA. | |
7. Ethical issues | Issues of privacy, confidentiality, informed consent, surveillance, and labelling students at risk | |
8. Structural factors | Structural concerns: commercial use of data or business-like practices; material conditions (technology) in using LA, especially considering countries with less-developed economies; a heightening of accountability processes; increasing competition among institutions; promotion of social inequalities and other exclusionary practices (for example, MOOCs promoted by Western universities in poor countries). Also, financial, political, philosophical, epistemological and technical-mathematical aspects being characteristically absent |
Discussion
(i) What are the main publication patterns of papers on LA and higher education contained in Web of Science (WoScc) and SciELO journals?
(ii) Which are the main methodologies in use in conducting research on LA in higher education?
(iii) Do these publications have an educational focus as stated by Ferguson (2012)?
(iv) Are there challenges and concerns regarding LA in the literature not so far identified?
-
Guaranteeing that the methods and statistical techniques associated with LA are neutral-free and do not secrete a bias in relation to students’ learning.
-
Acknowledging that—if students are aware that they are being observed and their behaviours are being measured-the virtual learning environments may prompt students to develop strategic behaviours and distort generated data.
-
Accepting that data generated and collected in specific educational settings are not context-free and cannot be generalised and straightforwardly applied to other contexts, in different moments.
-
LA and its use for commercial aims, comparisons, and audit cultures: these challenges remind us that data can be manipulated and used with unintended consequences.
-
LA might reinforce learning inequalities and the North/South imbalance. In this perspective, LA might be seen as exercising power and be a hegemonic tool that reinforces inequalities between wealthy and poor countries.
Conclusion
Recommendations and limitations
-
The number of analysed papers is limited since it included only WoScc core collection and SciELO indexes. This decision was based on the fact that both WoScc and SciELO databases share the same Clarivate Analytics’ Web of Science platform so helping in standardising the search. Future research might also consider SCOPUS, other locally recognised indexations, and books.
-
The span of time for the search (2013–2019) is also limited. However, given that one of the arguments of this paper is based on a seminal paper by Ferguson published in 2012, it was considered that this span of time is appropriate.