Towards a Robust Thermal-Visible Heterogeneous Face Recognition Approach Based on a Cycle Generative Adversarial Network.

Nadir Kamel Benamara; Ehlem Zigh; Tarik Boudghene Stambouli; Mokhtar Keche

doi:10.9781/ijimai.2021.12.003

Authors

Nadir Kamel Benamara Université des Sciences et de la Technologie d'Oran Mohamed Boudiaf
Ehlem Zigh Institut National des Postes et Télécommunications
Tarik Boudghene Stambouli Université des Sciences et de la Technologie d'Oran Mohamed Boudiaf
Mokhtar Keche Université des Sciences et de la Technologie d'Oran Mohamed Boudiaf

DOI:

https://doi.org/10.9781/ijimai.2021.12.003

Keywords:

Deep Learning, Generative Adversarial Network, Face Detection, Thermal Sensor

Abstract

Security is a sensitive area that concerns all authorities around the world due to the emerging terrorism phenomenon. Contactless biometric technologies such as face recognition have grown in interest for their capacity to identify probe subjects without any human interaction. Since traditional face recognition systems use visible spectrum sensors, their performances decrease rapidly when some visible imaging phenomena occur, mainly illumination changes. Unlike the visible spectrum, Infrared spectra are invariant to light changes, which makes them an alternative solution for face recognition. However, in infrared, the textural information is lost. We aim, in this paper, to benefit from visible and thermal spectra by proposing a new heterogeneous face recognition approach. This approach includes four scientific contributions. The first one is the annotation of a thermal face database, which has been shared via Github with all the scientific community. The second is the proposition of a multi-sensors face detector model based on the last YOLO v3 architecture, able to detect simultaneously faces captured in visible and thermal images. The third contribution takes up the challenge of modality gap reduction between visible and thermal spectra, by applying a new structure of CycleGAN, called TV-CycleGAN, which aims to synthesize visible-like face images from thermal face images. This new thermal-visible synthesis method includes all extreme poses and facial expressions in color space. To show the efficacy and the robustness of the proposed TV-CycleGAN, experiments have been applied on three challenging benchmark databases, including different real-world scenarios: TUFTS and its aligned version, NVIE and PUJ. The qualitative evaluation shows that our method generates more realistic faces. The quantitative one demonstrates that the proposed TV -CycleGAN gives the best improvement on face recognition rates. Therefore, instead of applying a direct matching from thermal to visible images which allows a recognition rate of 47,06% for TUFTS Database, a proposed TV-CycleGAN ensures accuracy of 57,56% for the same database. It contributes to a rate enhancement of 29,16%, and 15,71% for NVIE and PUJ databases, respectively. It reaches an accuracy enhancement of 18,5% for the aligned TUFTS database. It also outperforms some recent state of the art methods in terms of F1-Score, AUC/EER and other evaluation metrics. Furthermore, it should be mentioned that the obtained visible synthesized face images using TV-CycleGAN method are very promising for thermal facial landmark detection as a fourth contribution of this paper.

Downloads

Download data is not yet available.

References

N. K. Benamara, M. Keche, M. Wellington, Z. Munyaradzi, “Securing E-payment Systems by RFID and Deep Facial Biometry,” in 2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA), Riyadh, Saudi Arabia, Apr. 2021, pp. 151–157.

S. Dargan, M. Kumar, “A comprehensive survey on the biometric recognition systems based on physiological and behavioral modalities,” Expert Systems with Applications, vol. 143, p. 113114, Apr. 2020.

M. Turk, A. Pentland, “Eigenfaces for Recognition,” Journal of Cognitive Neuroscience, vol. 3, pp. 71–86, Jan. 1991.

P. Belhumeur, J. Hespanha, D. Kriegman, “Eigenfaces vs. Fisherfaces: recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, pp. 711–720, July 1997.

L. Wiskott, J.-M. Fellous, N. Kuiger, C. von der Malsburg, “Face recognition by elastic bunch graph matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, pp. 775–779, July 1997.

B. Hamdan, K. Mokhtar, “Face recognition using Angular Radial Transform,” Journal of King Saud University - Com- puter and Information Sciences, vol. 30, pp. 141–151, Apr. 2018.

R. Shoja Ghiass, O. Arandjelović, A. Bendada, X. Maldague, “Infrared face recognition: A comprehensive review of methodologies and databases,” Pattern Recognition, vol. 47, pp. 2807–2824, Sept. 2014.

Mamta, M. Hanmandlu, “Robust authentication using the unconstrained infrared face images,” Expert Systems with Applications, vol. 41, pp. 6494– 6511, Oct. 2014.

M. Kanti Bhowmik, Kankan, S. Majumder, G. Majumder, A. Saha, A. Nath, D. Bhattacharjee, D. K. Basu, M. Nasipuri, “Thermal Infrared Face Recognition – A Biometric Identification Technique for Robust Security system,” in Reviews, Refinements and New Ideas in Face Recognition, P. Corcoran Ed., InTech, July 2011.

M. Akhloufi, A. Bendada, J.-C. Batsale, “State of the art in infrared face recognition,” Quantitative InfraRed Thermography Journal, vol. 5, pp. 3–26, June 2008.

G. Pan, L. Sun, Z. Wu, S. Lao, “Eyeblink-based Anti-Spoofing in Face Recognition from a Generic Webcamera,” in 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, 2007, pp. 1–8, IEEE.

S. Jia, G. Guo, Z. Xu, “A survey on 3D mask presentation attack detection and countermeasures,” Pattern Recognition, vol. 98, p. 107032, Feb. 2020.

B. Hamdan, K. Mokhtar, “A self-immune to 3D masks attacks face recognition system,” Signal, Image and Video Processing, vol. 12, pp. 1053–1060, Sept. 2018.

S. Hu, N. Short, B. S. Riggan, M. Chasse, M. S. Sarfraz, “Het-erogeneous Face Recognition: Recent Advances in Infrared-to-Visible Matching,” in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, DC, USA, May 2017, pp. 883–890, IEEE.

R. Shoja Ghiass, H. Bendada, X. Maldague, “Université Laval Face Motion and Time-Lapse Video Database (UL-FMTV),” in Proceedings of the 2018 International Conference on Quantitative InfraRed Thermography, 2018, QIRT Council.

T. Bourlai, A. Ross, C. Chen, L. Hornak, “A study on using mid-wave infrared images for face recognition,” Baltimore, Maryland, May 2012, pp. 83711K–83711K–13.

T. Bourlai Ed., Face Recognition Across the Imaging Spectrum. Cham: Springer International Publishing, 2016.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, Koschan, M. Yi, M. A. Abidi, “Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition,” International Journal of Computer Vision, vol. 71, pp. 215–233, Feb. 2007.

D. Bhattacharjee, “Adaptive polar transform and fusion for human face image processing and evaluation,” Human- centric Computing and Information Sciences, vol. 4, p. 4, Dec. 2014.

A. R. Pal, A. Singha, “A comparative analysis of visual and thermal face image fusion based on different wavelet family,” in 2017 International Conference on Innovations in Electronics, Signal Processing and Communication (IESC), Shillong, India, Apr. 2017, pp. 213–218, IEEE.

G. Hermosilla, F. Gallardo, G. Farias, C. Martin, “Fusion of Visible and Thermal Descriptors Using Genetic Algo- rithms for Face Recognition Systems,” Sensors, vol. 15, pp. 17944–17962, July 2015.

N. K. Benamara, E. Zigh, T. Boudghene Stambouli, M. Keche, “Combined and Weighted Features for Robust Multispectral Face Recognition,” in Computational Intelligence and Its Applications, vol. 522, 2018, pp. 549–560.

N. K. Benamara, E. Zigh, T. B. Stambouli, M. Keche, “Efficient Multispectral Face Recognition using Random Feature Selection and PSOSVM,” in Proceedings of the 2nd International Conference on Networking, Information Systems & Security - NISS19, Rabat, Morocco, 2019, pp. 1–6.

K. Guo, S. Wu, Y. Xu, “Face recognition using both visible light image and near-infrared image and a deep network,” CAAI Transactions on Intelligence Technology, vol. 2, pp. 39–47, Mar. 2017.

D. Lin, X. Tang, “Inter-modality Face Recognition,” in Computer Vision – ECCV 2006, vol. 3954, A. Leonardis, H. Bischof, A. Pinz Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 13–26.

D. Yi, R. Liu, R. Chu, Z. Lei, S. Z. Li, “Face Matching Between Near Infrared and Visible Light Images,” in Advances in Biometrics, vol. 4642, S.-W. Lee, S. Z. Li Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 523–530.

F. Juefei-Xu, D. K. Pal, M. Savvides, “NIR-VIS heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, June 2015, pp. 141–150, IEEE.

S. Liu, D. Yi, Z. Lei, S. Z. Li, “Heterogeneous face image matching using multi-scale features,” in 2012 5th IAPR International Conference on Biometrics (ICB), New Delhi, India, Mar. 2012, pp. 79–84, IEEE.

L. Huang, J. Lu, Y.-P. Tan, “Learning modality-invariant features for heterogeneous face recognition,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Nov. 2012, pp. 1683–1686.

J. Lezama, Q. Qiu, G. Sapiro, “Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-Spectral Hallucination and Low-Rank Embedding,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, July 2017, pp. 6807–6816, IEEE.

K. Mallat, N. Damer, F. Boutros, A. Kuijper, J.-L. Dugelay, “Cross-spectrum thermal to visible face recognition based on cascaded image synthesis,” in 2019 International Conference on Biometrics (ICB), Crete, Greece, June 2019, pp. 1–8, IEEE.

A. Kantarcı, H. K. Ekenel, “Thermal to Visible Face Recognition Using Deep Autoencoders,” arXiv:2002.04219 [cs, eess], Feb. 2020.

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, “Gener- ative Adversarial Networks,” arXiv:1406.2661 [cs, stat], June 2014.

L. Song, M. Zhang, X. Wu, R. He, “Adversarial Discriminative Heterogeneous Face Recognition,” arXiv:1709.03675 [cs], Sept. 2017.

C. Chen, A. Ross, “Matching Thermal to Visible Face Images Using a Semantic-Guided Generative Adversarial Network,” in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France, May 2019, pp. 1–8, IEEE.

T. Zhang, A. Wiliem, S. Yang, B. Lovell, “TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition,” in 2018 International Conference on Biometrics (ICB), Gold Coast, QLD, Feb. 2018, pp. 174–181, IEEE.

W.-T. Chu, P.-S. Huang, “Thermal Face Recognition Based on Multi-scale Image Synthesis,” in MultiMedia Modeling, vol. 12572, J. Lokoč, T. Skopal, K. Schoeffmann, V. Mezaris, X. Li, S. Vrochidis, I. Patras Eds., Cham: Springer Interna- tional Publishing, 2021, pp. 99–110.

X. Di, B. S. Riggan, S. Hu, N. J. Short, V. M. Patel, “Multi-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 3, pp. 266–280, Apr. 2021.

H. Zhang, V. M. Patel, B. S. Riggan, S. Hu, “Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces,” in 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, Oct. 2017, pp. 100–107, IEEE.

B. S. Riggan, N. J. Short, S. Hu, “Thermal to Visible Synthesis of Face Images Using Multiple Regions,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, Mar. 2018, pp. 30–38, IEEE.

Z.-Q. Zhao, P. Zheng, S.-T. Xu, X. Wu, “Object Detection With Deep Learning: A Review,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, pp. 3212–3232, Nov. 2019.

R. Girshick, J. Donahue, T. Darrell, J. Malik, “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, June 2014, pp. 580–587, IEEE.

R. Girshick, “Fast R-CNN,” in 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, Dec. 2015, pp. 1440–1448, IEEE.

S. Ren, K. He, R. Girshick, J. Sun, “Faster R-CNN: To- wards Real-Time Object Detection with Region Proposal Networks,” arXiv:1506.01497 [cs], Jan. 2016.

J. Redmon, S. Divvala, R. Girshick, A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 2016, pp. 779–788, IEEE.

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, “SSD: Single Shot MultiBox Detector,” in Computer Vision – ECCV 2016, vol. 9905, B. Leibe, J. Matas, N. Sebe, M. Welling Eds., Cham: Springer International Publishing, 2016, pp. 21–37.

J. Redmon, A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv:1804.02767 [cs], Apr. 2018.

A. Kumar, A. Kaur, M. Kumar, “Face detection techniques: a review,” Artificial Intelligence Review, vol. 52, pp. 927–948, Aug. 2019.

H. Jiang, E. Learned-Miller, “Face Detection with the Faster R-CNN,” arXiv:1606.03473 [cs], June 2016.

R. Belaroussi, M. Milgram, “A comparative study on face detection and tracking algorithms,” Expert Systems with Applications, vol. 39, pp. 7158–7164, June 2012.

Y. K. Cheong, V. V. Yap, H. Nisar, “A novel face detection algorithm using thermal imaging,” in 2014 IEEE Symposium on Computer Applications and Industrial Electronics (ISCAIE), Penang, Malaysia, Apr. 2014, pp. 208–213, IEEE.

C. Ma, N. Trung, H. Uchiyama, H. Nagahara, A. Shimada, R.- Taniguchi, “Adapting Local Features for Face Detection in Thermal Image,” Sensors, vol. 17, p. 2741, Nov. 2017.

S. Yang, P. Luo, C. C. Loy, X. Tang, “WIDER FACE: A Face Detection Benchmark,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 2016, pp. 5525–5533, IEEE.

M. Mirza, S. Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784 [cs, stat], Nov. 2014.

P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, “Image-to- Image Translation with Conditional Adversarial Networks,” arXiv:1611.07004 [cs], Nov. 2018.

J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Net- works,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 2017, pp. 2242–2251, IEEE.

X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, S. P. Smolley, “Least Squares Generative Adversarial Networks,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 2017, pp. 2813–2821, IEEE.

K. Simonyan, A. Zisserman, “Very Deep Convolutional Net- works for Large-Scale Image Recognition,” arXiv:1409.1556 [cs], Apr. 2015.

O. M. Parkhi, A. Vedaldi, A. Zisserman, “Deep Face Recognition,” in Procedings of the British Machine Vision Conference 2015, Swansea, 2015, pp. 41.1–41.12, British Machine Vision Association.

K. He, X. Zhang, S. Ren, J. Sun, “Deep Residual Learning for Image Recognition,” arXiv:1512.03385 [cs], Dec. 2015.

Q. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman, “VGGFace2: A Dataset for Recognising Faces across Pose and Age,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, May 2018, pp. 67–74, IEEE.

O. Ronneberger, P. Fischer, T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, vol. 9351, N. Navab, J. Hornegger, W. M. Wells, F. Frangi Eds., Cham: Springer International Publishing, 2015, pp. 234–241.

K. Panetta, Q. Wan, S. Agaian, S. Rajeev, S. Kamath, R. Rajendran, S. P. Rao, A. Kaszowska, H. A. Taylor, A. Samani, X. Yuan, “A Comprehensive Database for Benchmarking Imaging Systems,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, pp. 509–520, Mar. 2020.

S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, X. Wang, “A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference,” IEEE Transactions on Multimedia, vol. 12, pp. 682–691, Nov. 2010.

S. Wang, Z. Liu, Z. Wang, G. Wu, P. Shen, S. He, X. Wang, “Analyses of a Multimodal Spontaneous Facial Expression Database,” IEEE Transactions on Affective Computing, vol. 4, pp. 34–46, Jan. 2013.

R. Pulecio, C. Gerardo, “Face recognition on distorted infrared images augmented by perceptual quiality-aware features,” S. Z. Li and A. K. Jain, Handbook of Face Recognition. New York, NY: Springer, 2005., Dec. 2016.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,” International Journal of Computer Vision, vol. 128, pp. 336–359, Feb. 2020.

M. Kopaczka, K. Acar, D. Merhof, “Robust Facial Landmark Detection and Face Tracking in Thermal Infrared Images using Active Appearance Models,” in Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Rome, Italy, 2016, pp. 150–158, SCITEPRESS - Science and Technology Publications.

D. Poster, S. Hu, N. Nasrabadi, B. Riggan, “An Examination of Deep-Learning Based Landmark Detection Methods on Thermal Face Imagery,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, June 2019, pp. 980–987, IEEE.

W.-T. Chu, Y.-H. Liu, “Thermal Facial Landmark Detection by Deep Multi-Task Learning,” in 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), Kuala Lumpur, Malaysia, Sept. 2019, pp. 1–6, IEEE.