Supporting Skill Assessment in Learning Experiences Based on Serious Games Through Process Mining Techniques.
DOI:
https://doi.org/10.9781/ijimai.2023.05.002Keywords:
Educational Data Mining, Learning Analytics, Game-Based Learning, Serious GamesAbstract
Learning experiences based on serious games are employed in multiple contexts. Players carry out multiple interactions during the gameplay to solve the different challenges faced. Those interactions can be registered in logs as large data sets providing the assessment process with objective information about the skills employed. Most assessment methods in learning experiences based on serious games rely on manual approaches, which do not scalewell when the amount of data increases. We propose an automated method to analyse students’ interactions and assess their skills in learning experiences based on serious games. The method takes into account not only the final model obtained by the student, but also the process followed to obtain it, extracted from game logs. The assessment method groups students according to their in-game errors and ingame outcomes. Then, the models for the most and the least successful students are discovered using process mining techniques. Similarities in their behaviour are analysed through conformance checking techniques to compare all the students with the most successful ones. Finally, the similarities found are quantified to build a classification of the students’ assessments. We have employed this method with Computer Science students playing a serious game to solve design problems in a course on databases. The findings show that process mining techniques can palliate the limitations of skill assessment methods in game-based learning experiences.
Downloads
References
D. Djaouti, J. Alvarez, J.-P. Jessel, O. Rampnoux, “Origins of Serious Games,” in Serious Games and Edutainment Applications, M. Ma, A. Oikonomou, L. C. Jain Eds., London: Springer London, 2011, pp. 25–43, doi: 10.1007/978-1-4471-2161-9_3.
T. M. Connolly, E. A. Boyle, E. Macarthur, T. Hainey, J. M. Boyle, “A systematic literature review of empirical evidence on computer games and serious games,” Computers & Education, vol. 59, pp. 661–686, 2012, doi: 10.1016/j.compedu.2012.03.004.
J. A. Caballero-Hernández, M. Palomo-Duarte, J. M. Dodero, “Skill assessment in learning experiences based on serious games: A Systematic Mapping Study,” Computers and Education, vol. 113, pp. 42–60, oct 2017, doi: 10.1016/j.compedu.2017.05.008.
G. Siemens, S. Dawson, G. Lynch, “Improving the Quality and Productivity of the Higher Education Sector Policy and Strategy for Systems-Level Deployment of Learning Analytics,” Society for Learning Analytics Research, 2013.
W. van der Aalst, A. Adriansyah, A. K. A. De Medeiros, F. Arcieri, T. Baier, T. Blickle, J. C. Bose, P. Van Den Brand, R. Brandtjen, J. Buijs, A. Burattin, J. Carmona, M. Castellanos, J. Claes, J. Cook, N. Costantini, F. Curbera, E. Damiani, M. De Leoni, ... M. Wynn, “Process mining manifesto,” in Lecture Notes in Business Information Processing, vol. 99 LNBIP, 2012, pp. 169–194, Springer Verlag.
T. Hainey, T. M. Connolly, Y. Chaudy, E. Boyle, R. Beeby, M. Soflano, “Assessment integration in serious games,” in Psychology, Pedagogy, and Assessment in Serious Games, IGI Global, nov 2013, pp. 317–341, doi: 10.4018/978-1-4666-4773-2.ch015.
M. Allen, Assessing academic programs in higher education. Bolton, MA: Anker, 2004.
V. J. Shute, “Stealth Assessment in Computer-Based Games To Support Learning,” in Computer Games and Instruction, S. Tobias, J. D. Fletcher Eds., Cambridge: MIT Press, 2011, ch. 20, pp. 503–524.
R. J. Mislevy, G. D. Haertel, “Implications of evidence- centered design for educational testing,” Educational Measurement: Issues and Practice, vol. 25, no. 4, pp. 6–20, 2006, doi: https://doi.org/10.1111/j.1745-3992.2006.00075.x
R. J. Mislevy, R. G. Almond, J. F. Lukas, “A brief introduction to evidencecentered design,” ETS Research Report Series, vol. 2003, no. 1, pp. i–29, 2003, doi: 10.1002/j.2333-8504.2003.tb01908.x.
V. J. Shute, M. Ventura, M. Bauer, D. Zapata-Rivera, “Melding the power of serious games and embedded assessment to monitor and foster learning,” in Serious games: Mechanisms and effects, U. Ritterfeld, M. J. Cody, P. Vorderer Eds., Routledge, 2009, pp. 295–321.
V. J. Shute, Y. J. Kim, “Formative and Stealth Assessment,” in Handbook of Research on Educational Communications and Technology, J. M. Spector, M. D. Merrill, J. Elen, M. J. Bishop Eds., New York, NY: Springer New York, 2014, pp. 311–321, doi: 10.1007/978-1-4614-3185-5_25.
A. Mitrovic, M. Mayo, P. Suraweera, B. Martin, “Constraint-Based Tutors: A Success Story,” in Engineering of Intelligent Systems, Berlin, Heidelberg, 2001, pp. 931–940, Springer Berlin Heidelberg.
P. Suraweera, A. Mitrovic, “An Intelligent Tutoring System for Entity Relationship Modelling,” International Journal of Artificial Intelligence in Education, vol. 14, pp. 375–417, 2004.
L. Zhuhadar, S. Marklin, E. Thrasher, M. D. Lytras, “Is there a gender difference in interacting with intelligent tutoring system? Can Bayesian Knowledge Tracing and Learning Curve Analysis Models answer this question?,” Computers in Human Behavior, vol. 61, pp. 198–204, 2016, doi: https://doi.org/10.1016/j.chb.2016.02.073
M. V. Yudelson, K. R. Koedinger, G. J. Gordon, “Individualized Bayesian Knowledge Tracing Models,” in Artificial Intelligence in Education, Berlin, Heidelberg, 2013, pp. 171–180, Springer Berlin Heidelberg.
W. M. P. van der Aalst, Process Mining Data Science in Action. Berlin Heidelberg: Springer, 2nd ed. ed., 2016. [18] K. Engelmann, M. Bannert, “Analyzing temporal data for understanding the learning process induced by metacognitive prompts,” Learning and Instruction, p. 101205, 2019, doi: https://doi.org/10.1016/j.learninstruc.2019.05.002
H. A. V. D. Berg, “Occam’s razor: From ockham’s via moderna to modern data science,” Science Progress, vol. 101, no. 3, pp. 261–272, 2018, doi: 10.3184/003685018X15295002645082.
M. de Leoni, W. M. van der Aalst, M. Dees, “A general process mining framework for correlating, predicting and clustering dynamic behavior based on event logs,” Information Systems, vol. 56, no. July, pp. 235–257, 2016, doi: 10.1016/j.is.2015.07.003.
A. Bogarín, R. Cerezo, C. Romero, “A survey on educational process mining,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, p. e1230, jan 2018, doi: 10.1002/widm.1230.
A. Bolt, M. de Leoni, W. M. P. van der Aalst, P. Gorissen, “Exploiting Process Cubes, Analytic Workflows and Process Mining for Business Process Reporting: A Case Study in Education,” in International Symposium on Data-driven Process Discovery and Analysis (SIMPDA), Vienna, Austria, 2015, pp. 33–47.
J. A. Caballero-Hernández, A. Balderas, M. Palomo- Duarte, P. Delatorre, A. J. Reinoso, J. M. Dodero, “Teamwork assessment in collaborative projects through process mining techniques,” International journal of engineering education, vol. 36, no. 1, pp. 470–482, 2020.
P. Reimann, “Time is precious: Variable-and event-centred approaches to process analysis in CSCL research,” International Journal of ComputerSupported Collaborative Learning, vol. 4, no. 3, pp. 239–257, 2009, doi: 10.1007/s11412-009-9070-z.
M. Bannert, P. Reimann, C. Sonnenberg, “Process mining techniques for analysing patterns and strategies in students’ self-regulated learning,” Metacognition and Learning, vol. 9, no. 2, pp. 161–185, 2014, doi: 10.1007/s11409-013-9107-6.
P. Reimann, L. Markauskaite, M. Bannert, “e-Research and learning theory: What do sequence and process mining methods contribute?,” British Journal of Educational Technology, vol. 45, no. 3, pp. 528–540, 2014, doi: 10.1111/bjet.12146.
H. M. W. Verbeek, J. C. A. M. Buijs, B. F. Van Dongen, W. M. van der Aalst, “ProM: The Process Mining Toolkit,” in International Conference on Business Process Management Demonstration Track, Hoboken, New Jersey, 2010, pp. 34–39.
V. U. Kumar, A. Krishna, P. Neelakanteswara, C. Z. Basha, “Advanced prediction of performance of a student in an university using machine learning techniques,” in 2020 International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India, 2020, pp. 121–126.
S. A. Alasadi, W. S. Bhaya, “Review of data preprocessing techniques in data mining,” Journal of Engineering and Applied Sciences, vol. 12, no. 16, pp. 4102–4107, 2017, doi: 10.3923/jeasci.2017.4102.4107.
S. J. J. Leemans, D. Fahland, W. M. P. van der Aalst, “Discovering BlockStructured Process Models from Event Logs Containing Infrequent Behaviour,” in Business Process Management Workshops. BPM 2013. Lecture Notes in Business Information Processing, vol 171., N. Lohmann, M. Song, P. Wohed Eds., Springer, Cham, 2014, pp. 66–78, doi: 10.1007/978-3-319-06257-0_6.
A. Rozinat, W. M. P. van der Aalst, “Conformance checking of processes based on monitoring real behavior,” Information Systems, vol. 33, no. 1, pp. 64–95, 2008, doi: https://doi.org/10.1016/j.is.2007.07.001
W. van der Aalst, A. Adriansyah, B. van Dongen, “Replaying history on process models for conformance checking and performance analysis,” WIREs Data Mining and Knowledge Discovery, vol. 2, no. 2, pp. 182– 192, 2012, doi: 10.1002/widm.1045.
C. Argyris, D. A. Schön, “Participatory action research and action science compared: A commentary,” American Behavioral Scientist, vol. 32, no. 5, pp. 612–623, 1989.
G. E. Mills, Action research: A guide for the teacher researcher. Boston: Pearson, 4th ed., 2011.
ACM, IEEE, “Computer Engineering Curricula 2016,” ACM, IEEE, 2016.
J. A. Caballero-Hernández, “Supporting skill assessment in learning experiences based on serious games through process mining techniques,” 2020. doi: 10.6084/m9.figshare.c.4916412.
A. Silberschatz, H. F. Korth, S. Sudarshan, Database system concepts. New York: McGraw-Hill, 6th ed. ed., 2011.
Downloads
Published
-
Abstract192
-
PDF29
Submit Article
Information
