Analysis of Gender Differences in Facial Expression Recognition Based on Deep Learning Using Explainable Artificial Intelligence.

Cristina Manresa Yee; Silvia Ramis; José M. Buades

doi:10.9781/ijimai.2023.04.003

Authors

Cristina Manresa Yee Universitat de les Illes Balears.
Silvia Ramis Universitat de les Illes Balears.
José M. Buades Universitat de les Illes Balears

DOI:

https://doi.org/10.9781/ijimai.2023.04.003

Keywords:

Explainable Artificial Intelligence, Facial Expression, Gender Differences, XAI

Supporting Agencies

This work has been supported by the Agencia Estatal de Investigación, project PID2019-104829RA-I00 / MCIN/ AEI / 10.13039/501100011033, EXPLainable Artificial INtelligence systems for health and well-beING (EXPLAINING).

Abstract

Potential uses of automated Facial Expression Recognition (FER) cover a wide range of applications such as customer behavior analysis, healthcare applications or providing personalized services. Data for machine learning play a fundamental role, therefore, understanding the relevancy of the data in the outcomes is of utmost importance. In this work we present a study on how gender influences the learning of a FER system. We analyze with Explainable Artificial intelligence (XAI) techniques how gender contributes to the learning and assess which facial expressions are more similar regarding face regions that impact on the classification.
Results show that there exist common regions in some expressions both for females and males with different intensities (e.g. happiness); however, there are other expressions like disgust, where important face regions differ. The insights of this work will help improving FER systems and understand the source of any inequality.

Downloads

Download data is not yet available.

References

D. McDuff, E. Kodra, R. el Kaliouby, and M. LaFrance, “A large-scale analysis of sex differences in facial expressions,” PLOS ONE, vol. 12, no. 4, pp. 1–11, 2017.

L. Cattaneo, V. Veroni, S. Boria, G. Tassinari, and L. Turella, “Sex Differences in Affective Facial Reactions Are Present in Childhood,” Frontiers in Integrative Neuroscience, vol. 12, 2018.

U. Dimberg and L.-O. Lundquist, “Gender differences in facial reactions to facial expressions.,” Biological Psychology, vol. 30, no. 2. Elsevier Science, Netherlands, pp. 151–159, 1990.

M. A. Kraines, L. J. A. Kelberer, and T. T. Wells, “Sex differences in attention to disgust facial expressions,” Cognition and Emotion, vol. 31, no. 8, pp. 1692–1697, 2017.

K. AM and G. AH., “Sex differences in emotion: expression, experience, and physiology,” J Pers Soc Psychol., vol. 74, no. 3, pp. 686–703, 1998.

M. Thunberg and U. Dimberg, “Gender Differences in Facial Reactions to Fear-Relevant Stimuli,” Journal of Nonverbal Behavior, vol. 24, no. 1, pp. 45–51, 2000.

G. E. Schwartz, S. ‐L Brown, and G. L. Ahern, “Facial Muscle Patterning and Subjective Experience During Affective Imagery: Sex Differences.,” Psychophysiology, vol. 17, pp. 75–82, 1980.

C. Evers, A. H. Fischer, and A. S. R. Manstead, “Gender and emotion regulation: a social appraisal perspective on anger.,” in Emotion regulation and well-being., Evers, Catharine: Department of Clinical and Health Psychology, Utrecht University, P.O. Box 80140, Utrecht, Netherlands, 3508 TC, c.evers@uu.nl: Springer Science + Business Media, 2011, pp. 211–222.

P. Ekman, “Universals and cultural differences in facial expressions of emotion,” Nebraska Symposium on Motivation, vol. 19, pp. 207–283, 1971.

P. Ekman and W. . Friesen, Facial action coding system: manual. Palo Alto, Calif.: Consulting Psychologists Press. OCLC: 5851545, 1978.

Y. Fan, J. C. K. Lam, and V. O. K. Li, “Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial action units of happiness,” Scientific Reports, vol. 11, no. 1, p. 5214, 2021.

O. Houstis and S. Kiliaridis, “Gender and age differences in facial expressions,” European Journal of Orthodontics, vol. 31, no. 5, pp. 459–466, 2009.

T. S. H. Wingenbach, C. Ashwin, and M. Brosnan, “Sex differences in facial emotion recognition across varying expression intensity levels from videos,” PloS one, vol. 13, no. 1, pp. e0190634–e0190634, Jan. 2018.

B. Montagne, R. P. C. Kessels, E. Frigerio, E. H. F. de Haan, and D. I. Perrett, “Sex differences in the perception of affective facial expressions: Do men really lack emotional sensitivity?,” Cognitive Processing, vol. 6, no. 2, pp. 136–141, 2005.

R. Campbell et al., “The classification of ‘fear’ from faces is associated with face recognition skill in women,” Neuropsychologia, vol. 40, no. 6, pp. 575–584, 2002.

A. K. Vail, J. F. Grafsgaard, K. E. Boyer, E. N. Wiebe, and J. C. Lester, “Gender Differences in Facial Expressions of Affect During Learning,” in Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, 2016, pp. 65–73.

J. C. Borod, E. Koff, and B. White, “Facial asymmetry in posed and spontaneous expressions of emotion,” Brain and Cognition, vol. 2, no. 2, pp. 165–175, 1983.

W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: review and insights,” Procedia Computer Science, vol. 175, pp. 689–694, 2020.

A. Domnich and G. Anbarjafari, “Responsible AI: Gender bias assessmentin emotion recognition,” arXiv, pp. 1–19, 2021.

M. Deramgozin, S. Jovanovic, H. Rabah, and N. Ramzan, A Hybrid Explainable AI Framework Applied to Global and Local Facial Expression Recognition. 2021.

T. Xu, J. White, S. Kalkan, and H. Gunes, “Investigating Bias and Fairness in Facial Expression Recognition,” in ECCV Workshops 2020, 2020.

Z. Wang et al., “Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8916-8925., 2020.

M. Lyons, M. Kamachi, and J. Gyoba, “The Japanese Female Facial Expression (JAFFE) Dataset.” Zenodo, 1998.

A. Heimerl, K. Weitz, T. Baur, and E. Andre, “Unraveling ML Models of Emotion with NOVA: Multi-Level Explainable AI for Non-Experts,” IEEE Transactions on Affective Computing, vol. 1, no. 1, pp. 1–13, 2020.

D. Schiller, T. Huber, M. Dietz, and E. André, “Relevance-Based Data Masking: A Model-Agnostic Transfer Learning Approach for Facial Expression Recognition,” Frontiers in Computer Science, vol. 2, p. 6, 2020.

P. Prajod, D. Schiller, T. Huber, and E. Andr’e, “Do Deep Neural Networks Forget Facial Action Units? - Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition,” ArXiv, vol. abs/2104.0, 2021.

K. Weitz, T. Hassan, U. Schmid, and J.-U. Garbas, “Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods,” Technisches Messen, vol. 86, no. 7–8, pp. 404–412, 2019.

G. del Castillo Torres, M. F. Roig-Maimó, M. Mascaró-Oliver, E. Amengual-Alcover, and R. Mas-Sansó, “Understanding How CNNs Recognize Facial Expressions: A Case Study with LIME and CEM,” Sensors, vol. 23, no. 1, 2023.

M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144.

M. Alber et al., “iNNvestigate Neural Networks!,” Journal of Machine Learning Research, vol. 20, no. 93, pp. 1–8, 2019.

S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation,” PLOS ONE, vol. 10, no. 7, pp. 1–46, 2015.

C. Manresa-Yee and S. Ramis, “Assessing Gender Bias in Predictive Algorithms Using EXplainable AI,” in Proceedings of the XXI International Conference on Human Computer Interaction, 2021.

K. Grabowski et al., “Emotional expression in psychiatric conditions: New technology for clinicians,” Psychiatry and Clinical Neurosciences, vol. 73, no. 2, pp. 50–62, 2019.

A. M. Barreto, “Application of facial expression studies on the field of marketing,” Emotional expression: the brain and the face, vol. 9, no. June, pp. 163–189, 2017.

S. Medjden, N. Ahmed, and M. Lataifeh, “Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor,” PLoS ONE, vol. 15, no. 7, p. e0235908, 2020.

A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,” IEEE Transactions on Affective Computing, vol. 10, no. 01, pp. 18–31, 2019.

P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992.

S. Li and W. Deng, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 356–370, 2019.

World Health Organization, “Gender,” Available online: https://www.who.int/europe/health-topics/gender. .

Y. Chen and J. Joo, “Understanding and Mitigating Annotation Bias in Facial Expression Recognition,” in ICCV 2021, 2021.

J.-L. Lisani, S. Ramis, and F. Perales, “A Contrario Detection of Faces: A Case Example,” SIAM Journal on Imaging Sciences, vol. 10, pp. 2091–2118, Jan. 2017.

C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge,” in 2013 IEEE International Conference on Computer Vision Workshops, 2013, pp. 397–403.

S. Ramis, J. . Buades, F. J. Perales, and C. Manresa-Yee, “A Novel Approach to Cross dataset studies in Facial Expression Recognition,” Multimedia Tools and Applications.

A. Barredo Arrieta et al., “Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, no. October 2019, pp. 82–115, 2020.

J. Ward, “Hierarchical Grouping to Optimize an Objective Function,” Journal of the American Statistical Association, vol. 58, no. 301, pp. 236–244, 1963.

O. Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.

C. Szegedy et al., “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.

M. Lin, C. Qiang, and Y. Shuicheng, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.

G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning with Applications in R. Springer, New York, 2013.

D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), 2015.

K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 6896–6905, 2020.

Q. T. Ngo and S. Yoon, “Facial expression recognition based on weighted-cluster loss and deep transfer learning using a highly imbalanced dataset,” Sensors (Switzerland), vol. 20, no. 9, 2020.

C.-T. Yen and K.-H. Li, “Discussions of Different Deep Transfer Learning Models for Emotion Recognitions,” IEEE Access, vol. 10, pp. 102860–102875, 2022.

H. G. Wallbott, “Big girls don’t frown, big boys don’t cry—Gender differences of professional actors in communicating emotion via facial expression.,” Journal of Nonverbal Behavior, vol. 12, no. 2, pp. 98–106, 1988.

N. K. Benamara, E. Zigh, T. B. Stambouli, and M. Keche, “Towards a Robust Thermal-Visible Heterogeneous Face Recognition Approach Based on a Cycle Generative Adversarial Network,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 4, pp. 132–145, 2022.

M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino, and J. Torresen, “A facial expression recognition system using robust face features from depth videos and deep learning,” Computers & Electrical Engineering, vol. 63, pp. 114–125, 2017.

A. Alcaide, M. A. Patricio, A. Berlanga, A. Arroyo, and J. J. Cuadrado-Gallego, “LIPSNN: A Light Intrusion-Proving Siamese Neural Network Model for Facial Verification,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 4, pp. 121–131, 2022.