A Systematic Literature Review of Empirical Studies on Learning Analytics in Educational Games.
DOI:
https://doi.org/10.9781/ijimai.2021.03.003Keywords:
Learning Analytics, Education, Data Mining, Games, Literature ReviewAbstract
Learning analytics (LA) in educational games is considered an emerging practice due to its potential of enhancing the learning process. Growing research on formative assessment has shed light on the ways in which students' meaningful and in-situ learning experiences can be supported through educational games. To understand learners' playful experiences during gameplay, researchers have applied LA, which focuses on understanding students' in-game behaviour trajectories and personal learning needs during play. However, there is a lack of studies exploring how further research on LA in educational games can be conducted. Only a few analyses have discussed how LA has been designed, integrated, and implemented in educational games. Accordingly, this systematic literature review examined how LA in educational games has evolved. The study findings suggest that: (1) there is an increasing need to consider factors such as student modelling, iterative game design and personalisation when designing and implementing LA through educational games; and (2) the use of LA creates several challenges from technical, data management and ethical perspectives. In addition to outlining these findings, this article offers important notes for practitioners, and discusses the implications of the study’s results.
Downloads
References
W.-C. Hsu and H.-C. K. Lin, “Impact of applying WebGL technology to develop a web digital game-based learning system for computer programming course in flipped classroom,” in International Conference on Educational Innovation through Technology (EITT), Tainan, Taiwan, China, 2016, doi: 10.1109/EITT.2016.20.
F. Ke., “An implementation of design-based learning through creating educational computer games: A case study on mathematics learning during design and computing,” Computers & Education, vol. 73, pp. 26-39, 2014, doi: 10.1016/j.compedu.2013.12.010.
T.-Y. Liu and Y.-L. Chu, “Using ubiquitous games in an English listening and speaking course: Impact on learning outcomes and motivation,” Computers & Education, vol. 55, no. 2, pp. 630-643, 2010, doi: 10.1016/j.compedu.2010.02.023.
S. Suh, S. W. Kim, and N. J. Kim, “Effectiveness of MMORPG‐based instruction in elementary English education in Korea,” Journal of Computer Assisted Learning, vol. 26, no. 5, pp. 370-378, 2010, doi: 10.1111/j.1365-2729.2010.00353.x.
V. J. Shute, L. Wang, S. Greiff, W. Zhao, and G. Moore, “Measuring problem solving skills via stealth assessment in an engaging video game,” Computers in Human Behavior, vol. 63, pp. 106-117, 2016, doi: 10.1016/j. chb.2016.05.047.
D. Burgos, C. Tattersall, and R. Koper, “Re-purposing existing generic games and simulations for e-learning,” Computers in Human Behavior, vol. 23, no. 6, pp. 2656-2667, 2006, doi: 10.1016/j.chb.2006.08.002.
P. Moreno-Ger and D. Burgos, “The case for serious games analytics,” in Radical Solutions and Learning Analytics: Personalised Learning and Teaching Through Big Data, D. Burgos (Ed.), Singapore: Springer, 2020, pp. 213-227, doi: 10.1007/978-981-15-4526-9_13.
D. Burgos, Radical solutions and learning Analytics, Singapore: Springer, 2020.
H. Fournier, R. Kop, and G. Durand, “Challenges to research in MOOCs,” Journal of Online Learning and Teaching / MERLOT, vol. 10, no. 1, pp. 1–15, 2014.
J. Frommel, K. Rogers, J. Brich, D. Besserer, L. Bradatsch, I. Ortinau, ... and M. Weber, Integrated questionnaires: Maintaining presence in game environments for self-reported data acquisition, in Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, London, United Kingdom, 2015, pp. 359-368.
D. D. Reese, B.G. Tabachnick, and R. E. Kosko, “Video game learning dynamics: Actionable measures of multidimensional learning trajectories,” British Journal of Educational Technology, vol. 46, no. 1, pp. 98-122, 2015, doi: 10.1111/bjet.12128
V. J. Shute and M. Ventura, Stealth assessment: Measuring and supporting learning in video games, London, England: The MIT Press, 2013.
B. R. Belland, “The role of construct definition in the creation of formative assessments in game-based learning,” in Assessment in game-based learning, New York, United States: Springer, 2012, pp. 29-42, doi: 10.1007/978-1-4614-3546-4_3.
V. J. Shute, M. Ventura, M. Bauer, and D. Zapata-Rivera, “Melding the power of serious games and embedded assessment to monitor and foster learning,” in Serious games: Mechanisms and effects, New York, United States: Routledge, 2009, pp. 295-321.
Á. Serrano-Laguna, J. Torrente, P. Moreno-Ger, and B. Fernández-Manjón. “Application of LA in educational videogames,” Entertainment Computing, vol. 5, no. 4, pp. 313-322. 2014. doi: 10.1016/j.entcom.2014.02.003.
I. J. Perez-Colado, V. M. Perez-Colado, I. Martínez-Ortiz, M. FreireMoran, and B. Fernández-Manjón, “UAdventure: The eAdventure reboot: Combining the experience of commercial gaming tools and tailored educational tools,” in IEEE Global Engineering Education Conference (EDUCON), Athens, Greece, 2017.
A. Tlili, F. Essalmi, M. Jemni, Kinshuk, and N.S. Chen, “A complete validated learning analytics framework: Designing issues from data preparation perspective,” International Journal of Information and Communication Technology Education (IJICTE), vol. 14, no. 2, pp. 1–16, 2018, doi: 10.4018/IJICTE.2019070104.
H. Fournier, R. Kop, and H. Sitlia, “The value of learning analytics to networked learning on a personal learning environment,” in Proceedings of the 1st International Conference on Learning Analytics and Knowledge, Banff, Canada, 2011, pp. 104-109, doi:10.1145/2090116.2090131.
S. Powell and S. MacNeill, “Institutional readiness for analytics,” JISC CETIS Analytics Series, vol. 1, no. 8, 2012.
D. Gijbels and F. Dochy, “Students’ assessment preferences and approaches to learning: can formative assessment make a difference?,” Educational Studies, vol. 32, no. 4, pp. 399-409. 2006. doi: 10.1080/03055690600850354.
V. J. Shute and M. Ventura, “Stealth assessment,” in The SAGE Encyclopedia of Educational Technology, Thousand Oaks, United States: SAGE, 2015, pp. 675-678, doi: 10.4135/9781483346397.
C.C. van Nimwegen, D. Burgos, H.H. van Oostendorp and H.H. Schijf, “The paradox of the assisted user: guidance can be counterproductive” in Proceedings of the SIGCHI conference on Human Factors in computing systems, Montreal, Canada, 2006, pp.917-926.
C. Alonso-Fernandez, A. Calvo, M. Freire, I. Martinez-Ortiz, and B. Fernandez-Manjon. “Systematizing game learning analytics for serious games” in IEEE Global Engineering Education Conference (EDUCON), Athens, Greece, 2017, pp. 1111-1118.
A. Tlili and M. Chang, “Data analytics approaches in educational games and gamification Systems: Summary, challenges, and future insights,” in Data Analytics Approaches in Educational Games and Gamification Systems, Singapore: Springer, 2019, pp. 249-255, doi:10.1007/978-981-32-9335-9.
Z. K. Papamitsiou and A.A. Economides, “Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence,” Journal of Educational Technology & Society, vol. 17, pp. 49-64, 2014.
G. L. Saveski, W. Westera, L. Yuan, P. Hollins, B.F. Manjón, P.M. Ger, and K. Stefanov, “What serious game studios want from ICT research: identifying developers’ needs,” in International Conference on Games and Learning Alliance, Rome, Italy, 2015, pp. 32-41.
C. Okoli and K. Schabram, “A guide to conducting a systematic literature review of information systems research,” Sprouts: Working Papers on Information Systems, vol. 10, no. 26, 2010, doi: 10.2139/ssrn.1954824.
T. M. Connolly, E. A. Boyle, E. MacArthur, T. Hainey, and J. M. Boyle, “A systematic literature review of empirical evidence on computer games and serious games,” Computers & Education, vol. 59, no. 2, pp. 661-686, 2012, doi: 10.1016/j.compedu.2012.03.004.
J. Webster and R. T. Watson, “Analyzing the past to prepare for the future: Writing a literature review,” MIS Quarterly, vol. 26, no. 2, pp. 13-23, 2002.
M. Minović, M. Milovanović, U. Šošević, and M. Á. C. González. “Visualisation of student learning model in serious games,” Computers in Human Behavior, vol. 47, pp. 98-107, 2015, doi: 10.1016/j.chb.2014.09.005.
C. Alonso-Fernández, I. Martínez‐Ortiz, R. Caballero, M. Freire, and B. Fernández‐Manjón, “Predicting students’ knowledge after playing a serious game based on learning analytics data: A case study,” Journal of Computer Assisted Learning, vol. 36, no. 3, 2019, doi: 10.1111/jcal.12405.
M. T. Cheng, Y. W. Lin and H. C. She, “Learning through playing virtual Age: Exploring the interactions among student concept learning, gaming performance, in-game behaviors, and the use of in-game characters,” Computers & Education, vol. 86, pp. 18-29, 2015, doi: 10.1016/j.compedu.2015.03.007.
M. Denden, A. Tlili, F. Essalmi, and M. Jemni, “Implicit modeling of learners’ personalities in a game-based learning environment using their gaming behaviors,” Smart Learning Environments, vol. 5, no. 1, 2018, doi: 10.1186/s40561-018-0078-6.
F. Essalmi, A. Tlili, L. J. B. Ayed, and M. Jemni. “Toward modeling the learner’s personality using educational games,” International Journal of Distance Education Technologies (IJDET), vol. 15, no. 4, pp. 21-38, 2017.
C. M. Forsyth, A. C. Graesser, P. Pavlik Jr, Z. Cai, H. Butler, D. Halpern, and K. Millis. “Operation aries!: Methods, mystery, and mixed models: Discourse features predict affect in a serious game,” Journal of Educational Data Mining, vol. 5, no. 1, pp. 147-189, 2013, 10.5281/zenodo.3554615.
H. T. Hou, “Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG),” Computers & Education, vol. 58, no. 4, pp. 1225-1233, 2012, doi: 10.1016/j.compedu.2011.11.015.
R. Israel-Fishelson and A. Hershkovitz, “Persistence in a game-based learning environment: The case of elementary school students learning computational thinking,” Journal of Educational Computing Research, vol. 58, no. 5, pp. 891-918, 2019, doi: 10.1177/0735633119887187.
D. Kerr and G. K. Chung. “Identifying key features of student performance in educational video games and simulations through cluster analysis,” Journal of Educational Data Mining, vol. 4, no. 1, pp. 144-182, 2012, doi: 10.5281/zenodo.3554647.
M. A. Khenissi, F. Essalmi, M. Jemni, T. W. Chang, and N. S. Chen. “Unobtrusive monitoring of learners’ interactions with educational games for measuring their working memory capacity,” British Journal of Educational Technology, vol. 48, no. 2, pp. 224-245, 2017, doi: 10.1111/bjet.12445.
M. Liu, J. Lee, J. Kang, and S. Liu. “What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game,” Technology, Knowledge and Learning, vol. 21, no. 1, pp. 33-57, 2016, doi: 10.1007/s10758-015-9263- 7.
E. Lotfi, B. Amine, and B. Mohammed. “Players performances analysis based on educational data mining case of study: Interactive waste sorting serious game,” International Journal of Computer Applications, vol. 108, no. 11, pp. 13-19, 2014.
T. Martin, C. Petrick Smith, N. Forsgren, A. Aghababyan, P. Janisiewicz, and S. Baker. “Learning fractions by splitting: Using learning analytics to illuminate the development of mathematical understanding,” Journal of the Learning Sciences, vol. 24, no. 4, pp. 593-637, 2015, doi: 0.1080/10508406.2015.1078244.
C. T. Wen, C. J. Chang, M. H. Chang, S. H. F. Chiang, C. C. Liu, F. K. Hwang, and C. C. Tsai. “The learning analytics of model-based learning facilitated by a problem-solving simulation game,” Instructional Science, vol. 46, no. 6, pp. 847-867, 2018, doi:10.1007/s11251-018-9461-5.
G. Altanis, M. Boloudakis, S. Retalis, and N. Nikou. “Children with motor impairments play a kinect learning game: First findings from a pilot case in an authentic classroom environment,” Journal of Interact Design Architect, vol. 19, pp. 91-104, 2013.
A. Calvo-Morata, D. C. Rotaru, C. Alonso-Fernández, M. Freire, I. Martínez-Ortiz, and B. Fernández-Manjón. “Validation of a cyberbullying serious game using game analytics,” IEEE Transactions on Learning Technologies. vol. 13, no. 1, pp. 186-197, 2018.
A. R. Cano, B. Fernández‐Manjón, and Á. J. García‐Tejedor. “Using game learning analytics for validating the design of a learning game for adults with intellectual disabilities,” British Journal of Educational Technology, vol. 49, no. 4, pp. 659-672, 2018.
D. Kerr. “Using data mining results to improve educational video game design,” Journal of Educational Data Mining, vol. 7, no. 3, pp. 1-17, 2015.
Á. Serrano-Laguna, B. Manero, M. Freire, and B. Fernández-Manjón. “A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes,” Multimedia Tools and Applications, vol. 77, no. 2, pp. 2849-2871, 2018. doi: 10.1007/s11042-017-4467-6.
Y. Chaudy and T. Connolly. “Specification and evaluation of an assessment engine for educational games: Integrating learning analytics and providing an assessment authoring tool,” Entertainment Computing, vol. 30, 2019, doi: 10.31686/IJIER.VOL8.ISS3.2220.
M. Callaghan, N, McShane, A. G. Eguíluz, and M. Savin-Baden. “Extending the activity theory based model for serious games design in engineering to integrate analytics,” International Journal of Engineering Pedagogy (iJEP), vol. 8, no. 1, pp. 109-126, 10.3991/ijep.v8i1.8087.
Z. H. Chen and S. Y. Lee. “Application-driven educational game to assist young children in learning english vocabulary,” Journal of Educational Technology & Society, vol. 21, no. 1, pp. 70-81, 2018.
K. Kiili and H. Ketamo. “Evaluating cognitive and affective outcomes of a digital game-based math test,” IEEE Transactions on Learning Technologies, vol. 11, no. 2, pp. 255-263, 2018.
K. Kiili, K. Moeller, and M. Ninaus, “Evaluating the effectiveness of a game-based rational number training-In-game metrics as learning indicators,” Computers & Education, vol. 120, pp. 13-28, 2018, doi: 10.1016/j.compedu.2018.01.012.
D. Rodríguez-Cerezo, A. Sarasa-Cabezuelo, M. Gómez-Albarrán, and J. L. Sierra, “Serious games in tertiary education: A case study concerning the comprehension of basic concepts in computer language implementation courses,” Computers in Human Behavior, vol. 31, pp. 558-570, 2014, doi: 10.1016/j.chb.2013.06.009.
A. Capatina, G. Bleoju, E. Rancati, and E. Hoareau, “Tracking precursors of learning analytics over serious game team performance ranking,” Behaviour & Information Technology, vol. 37, no. 10-11, pp. 1008-1020, 2018, doi: 10.1080/0144929X.2018.1474949.
A. B. Hernández-Lara, A. Perera-Lluna, and E. Serradell-López. “Applying learning analytics to students’ interaction in business simulation games: The usefulness of learning analytics to know what students really learn,” Computers in Human Behavior. vol. 92, pp. 600-612. 2018, doi: 10.1016/j.chb.2018.03.001.
M. Kourakli, I. Altanis, S. Retalis, M. Boloudakis, D. Zbainos, and K. Antonopoulou. “Towards the improvement of the cognitive, motoric, and academic skills of students with special educational needs using Kinect learning games,” International Journal of Child-Computer Interaction, vol. 11, pp. 28-39, 2017, doi: 10.1016/j.ijcci.2016.10.009.
C. S. Loh and Y. Sheng. “Maximum similarity index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games,” Computers in Human Behavior, vol. 39, pp. 322-330, 2014, doi: 10.1016/j.chb.2014.07.022.
C. S. Loh and Y. Sheng. “Measuring the (dis-) similarity between expert and novice behaviors as serious games analytics,” Education and Information Technologies, vol. 20, no. 1, pp. 5-19, 2015, doi: 10.1007/s10639-013-9263-y.
C. S. Loh, Y. Sheng, and I. H. Li. “Predicting expert-novice performance as serious games analytics with objective-oriented and navigational action sequences,” Computers in Human Behavior, vol. 49, pp. 147-155, 2015. doi: 10.1016/j.chb.2015.02.053.
E. Rowe, J. Asbell-Clarke, R. S. Baker, M. Eagle, A. G. Hicks, T. M. Barnes, and T. Edwards. “Assessing implicit science learning in digital games,” Computers in Human Behavior, vol. 76, pp. 617-630, 2017, doi: 10.1016/j.chb.2017.03.043.
A. Slimani, F. Elouaai, L. Elaachak, O. B. Yedri, M. Bouhorma, and M. Sbert. “Learning analytics through serious games: Data mining algorithms for performance measurement and improvement purpose,” International Journal of Emerging Technologies in Learning (iJET), vol. 13, no. 1, pp. 46-64, 2018.
E. L. Snow, L. K. Allen, M. E. Jacovina, and D. S. McNamara. “Does agency matter?: Exploring the impact of controlled behaviors within a game-based environment,” Computers & Education, vol. 82, pp. 378-392, 2015. doi: 10.1016/j.compedu.2014.12.011.
K. Becker, “How are games educational? Learning theories embodied in games,” in DiGRA: Changing Views - Worlds in Play. Vancouver, Canada, 2005.
E. Z. F., Liu, and C. H. Lin. “Developing evaluative indicators for educational computer games,” British Journal of Educational Technology, vol. 40, no. 1, pp. 174-178, 2009, doi: 10.1111/j.1467-8535.2008.00852.x.
J. Moon and Z. Liu. “Rich Representations for Analyzing Learning Trajectories: Systematic Review on Sequential Data Analytics in Game-Based Learning Research,” in Data Analytics Approaches in Educational Games and Gamification Systems, Tlili A., Chang M. (eds), Singapore: Springer, 2019.
A. Tlili, F. Essalmi, and M. Jemni, “An educational game for teaching computer architecture: Evaluation using learing analytics,” in 5th IEEE International Conference on Information & Communication Technology and Accessibility (ICTA), Marrakech, Morocco, 2016, pp. 1-6.
A. Pardo and G. Siemens, “Ethical and privacy principles for learning analytics,” British Journal of Educational Technology, vol. 45, no. 3, pp. 438–450, 2014, doi: 10.1111/bjet.12152.
S. Welsh and S. McKinney, “Clearing the Fog: A learning Analytics Code of Practice,” in Proceedings of the Australasian Society for Computers in Learning in Tertiary Education, T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Perth, Australia. 2015.
S. Erhel and E. Jamet. “Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness,” Computers & Education, vol. 67, pp. 156-167, 2013, doi: 10.1016/j.compedu.2013.02.019.
Downloads
Published
-
Abstract284
-
PDF23
Submit Article
Information
