A Case-Based Reasoning Model Powered by Deep Learning for Radiology Report Recommendation.

Elvira Amador Domínguez; Emilio Serrano; Daniel Manrique; Javier Bajo

doi:10.9781/ijimai.2021.08.011

Authors

Elvira Amador Domínguez Universidad Politécnica de Madrid
Emilio Serrano Universidad Politécnica de Madrid
Daniel Manrique Universidad Politécnica de Madrid
Javier Bajo Universidad Politécnica de Madrid

DOI:

https://doi.org/10.9781/ijimai.2021.08.011

Keywords:

Case-Based Reasoning, Deep Learning, Natural Language Processing, Entity Recognition, Medical Radiology

Supporting Agencies

The authors thank the reviewers and editors for their valuable comments and suggestions, which have improved this paper. This project has received funding from the Horizon 2020 research and innovation programme of the European Union, under grant agreement No. 825619. This work has also been supported by the Autonomous Region of Madrid through the program CABAHLA-CM (GA No. P2018/TCS-4423) and by the “Universidad Politécnica de Madrid” under the program “Ayudas para Contratos Predoctorales para la Realización del Doctorado”. The authors would like to thank Jérémy Clech and Guillaume Martial from NEHS DIGITAL for their support and numerous comments during the development of this work.

Abstract

Case-Based Reasoning models are one of the most used reasoning paradigms in expert-knowledge-driven areas. One of the most prominent fields of use of these systems is the medical sector, where explainable models are required. However, these models are considerably reliant on user input and the introduction of relevant curated data. Deep learning approaches offer an analogous solution, where user input is not required. This paper proposes a hybrid Case-Based Reasoning, Deep Learning framework for medical-related applications, focusing on the generation of medical reports. The proposal combines the explainability and user-focused approach of case-based reasoning models with the deep learning techniques performance. Moreover, the framework is fully modular to fit a wide variety of tasks and data, such as real-time sensor captured data, images, or text, to name a few. An implementation of the proposed framework focusing on radiology report generation assistance is provided. This implementation is used to evaluate the proposal, showing that it can provide meaningful and accurate corrections, even when the amount of information available is minimal. Additional tests on the optimization degree of the case base are also performed, evidencing how the proposed framework can optimize this base to achieve optimal performance.

Downloads

Download data is not yet available.

References

Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao,T. Zhang, S. Gao, J. Liu, “Ce-net: Context encoder network for 2d medical image segmentation,” IEEE Transactions on Medical Imaging, vol. 38, no. 10, pp. 2281–2292, 2019, doi: 10.1109/TMI.2019.2903562.

D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. D. Lange, P. Halvorsen, H. D. Johansen, “Resunet++: An advanced architecture for medical image segmentation,” in 2019 IEEE International Symposium on Multimedia (ISM), San Diego, CA, USA, 2019, pp. 225–2255.

D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, H. D. Johansen, “Kvasir-seg: A segmented polyp dataset,” in MultiMedia Modeling, New York, New York, USA, 2020, pp. 451–462, Springer International Publishing.

A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, M. Niessner, “Scan2cad: Learning cad model alignment in rgb-d scans,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, California, USA, June 2019.

B. Yang, S. Wang, A. Markham, N. Trigoni, “Robust attentional aggregation of deep feature sets for multi-view 3d reconstruction,” International Journal of Computer Vision, vol. 128, pp. 53–73, Jan 2020, doi: 10.1007/s11263-019-01217-w.

J. Liu, Z. Zhang, N. Razavian, “Deep ehr: Chronic disease prediction using medical notes,” in Proceedings of the 3rd Machine Learning for Healthcare Conference, vol. 85 of Proceedings of Machine Learning Research, Palo Alto, California, 17–18 Aug 2018, pp. 440–464, PMLR.

S. Parisot, S. I. Ktena, E. Ferrante, M. Lee, R. Guerrero, B. Glocker, D. Rueckert, “Disease prediction using graph convolutional networks: Application to autism spectrum disorder and alzheimer’s disease,” Medical Image Analysis, vol. 48, p. 117–130, Aug 2018, doi: 10.1016/j.media.2018.06.001.

J. Kolodner, Case-Based Reasoning. San Francisco, California, USA: Morgan Kaufmann Publishers Inc., 1993.

M. M. Richter, R. O. Weber, Case-Based Reasoning: A Textbook. New York City, New York, USA: Springer Publishing Company, Incorporated, 2013.

G. Costa Silva, E. E. O. Carvalho, W. M. Caminhas, “An artificial immune systems approach to Case-based Reasoning applied to fault detection and diagnosis,” Expert Systems with Applications, vol. 140, p. 112906, Feb. 2020, doi: 10.1016/j.eswa.2019.112906.

F. Torrent-Fontbona, J. Massana, B. López, “Case-base maintenance of a personalised and adaptive CBR bolus insulin recommender system for type 1 diabetes,” Expert Systems with Applications, vol. 121, pp. 338–346, May 2019, doi: 10.1016/j.eswa.2018.12.036.

E. Amador-Domínguez, E. Serrano, D. Manrique, J. F. D. Paz, “Prediction and decision-making in intelligent environments supported by knowledge graphs, A systematic review,” Sensors, vol. 19, no. 8, p. 1774, 2019, doi: 10.3390/s19081774.

“Ai4eu.” https://www.ai4eu.eu/ Accessed: 2020-12-21.

“The ai4eu scientific vision.” https://www.ai4eu.eu/ai4eu-scientificvision Accessed: 2020-12-21.

“Ai4eu.” https://www.ai4eu.eu/resource/raidologist Accessed: 2020-12-21.

M. B. Bentaiba-Lagrid, L. Bouzar-Benlabiod, S. H. Rubin, T. Bouabana-Tebibel, M. R. Hanini, “A case-based reasoning system for supervised classification problems in the medical field,” Expert Systems with Applications, vol. 150, p. 113335, July 2020, doi: 10.1016/j.eswa.2020.113335.

D. Brown, A. Aldea, R. Harrison, C. Martin, I. Bayley, “Temporal casebased reasoning for type 1 diabetes mellitus bolus insulin decision support,” Artificial Intelligence in Medicine, vol. 85, pp. 28–42, Apr. 2018, doi: 10.1016/j.artmed.2017.09.007.

E. Lupiani, J. M. Juarez, J. Palma, R. Marin, “Monitoring elderly people at home with temporal Case-Based Reasoning,” Knowledge-Based Systems, vol. 134, pp. 116–134, oct 2017, doi: 10.1016/j.knosys.2017.07.025.

A. Aamodt, E. Plaza, “Case-based reasoning: Foundational issues, methodological variations, and system approaches,” AI Commun., vol. 7, p. 39–59, Mar. 1994, doi: 10.3233/AIC-1994-7104.

Y. Qin, W. Lu, Q. Qi, X. Liu, M. Huang, P. J. Scott, X. Jiang, “Towards an ontology-supported case-based reasoning approach for computer-aided tolerance specification,” Knowledge-Based Systems, vol. 141, pp. 129–147, Feb. 2018, doi: 10.1016/j.knosys.2017.11.013.

J. Daengdej, D. Lukose, R. Murison, “Using statistical models and case-based reasoning in claims prediction: experience from a real-world problem,” Knowledge-Based Systems, vol. 12, pp. 239–245, Oct. 1999, doi: 10.1016/S0950-7051(99)00015-5.

S. Nasiri, G. Zahedi, S. Kuntz, M. Fathi, “Knowledge representation and management based on an ontological CBR system for dementia caregiving,” Neurocomputing, vol. 350, pp. 181–194, jul 2019, doi: 10.1016/j.neucom.2019.04.027.

F. Marie, L. Corbat, Y. Chaussy, T. Delavelle, J. Henriet, J.-C. Lapayre, “Segmentation of deformed kidneys and nephroblastoma using CaseBased Reasoning and Convolutional Neural Network,” Expert Systems with Applications, vol. 127, pp. 282–294, Aug. 2019, doi: 10.1016/j.eswa.2019.03.010.

L. Corbat, M. Nauval, J. Henriet, J.-C. Lapayre, “A fusion method based on Deep Learning and Case-Based Reasoning which improves the resulting medical image segmentations,” Expert Systems with Applications, vol. 147, p. 113200, jun 2020, doi: 10.1016/j.eswa.2020.113200.

J.-B. Lamy, B. Sekar, G. Guezennec, J. Bouaud, B. Séroussi, “Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach,” Artificial Intelligence in Medicine, vol. 94, pp. 42–53, Mar. 2019, doi: 10.1016/j.artmed.2019.01.001.

V. Tang, K. Choy, G. Ho, H. Lam, Y. Tsang, “An iomt-based geriatric care management system for achieving smart health in nursing homes,” Industrial Management and Data Systems, vol. 119, no. 8, pp. 1819–1840, 2019, doi: 10.1108/IMDS-01-2019-0024.

S. Massie, G. Forbes, S. Craw, L. Fraser, G. Hamilton, “Fitsense: Employing multi-modal sensors in smart homes to predict falls,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11156 LNAI, pp. 249–263, 2018, doi: 10.1007/978-3-030-01081-2.

G. Forbes, “Employing multi-modal sensors for personalised smart home health monitoring,” vol. 2567, 2019, pp. 185–190.

G. Mujtaba, L. Shuib, N. Idris, W. L. Hoo, R. G. Raj, K. Khowaja, K. Shaikh, H. F. Nweke, “Clinical text classification research trends: Systematic literature review and open issues,” Expert Systems with Applications, vol. 116, pp. 494–520, 2019, doi: 10.1016/j.eswa.2018.09.034.

J. T. Oliva, J. L. G. Rosa, “Classification for EEG report generation and epilepsy detection,” Neurocomputing, vol. 335, pp. 81–95, 2019, doi: 10.1016/j.neucom.2019.01.053.

S. Baccianella, A. Esuli, F. Sebastiani, “Variable-constraint classification and quantification of radiology reports under the ACR Index,” Expert Systems with Applications, vol. 40, no. 9, pp. 3441–3449, 2013, doi: 10.1016/j.eswa.2012.12.052.

C. R. Olsen, R. J. Mentz, K. J. Anstrom, D. Page, P. A. Patel, “Clinical applications of machine learning in the diagnosis, classification, and prediction of heart failure,” American Heart Journal, pp. 1–17, 2020, doi: 10.1016/j.ahj.2020.07.009.

A. Dudchenko, M. Ganzinger, G. Kopanitsa, “Diagnoses Detection in Short Snippets of Narrative Medical Texts,” Procedia Computer Science, vol. 156, pp. 150 – 157, 2019, doi: 10.1016/j.procs.2019.08.190.

J. Prada, Y. Gala, A. Sierra, “Covid-19 mortality risk prediction using x-ray images,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 6, pp. 7–14, 2021, doi: 10.9781/ijimai.2021.04.001.

K. Negi, A. Pavuri, L. Patel, C. Jain, “A novel method for drug-adverse event extraction using machine learning,” Informatics in Medicine Unlocked, vol. 17, p. 100190, 2019, doi: 10.1016/j.imu.2019.100190.

“Detection of unexpected findings in radiology reports: A comparative study of machine learning approaches,” vol. 160, p. 113647, 2020, doi: 10.1016/j.eswa.2020.113647.

T. F. d. Toledo, H. D. Lee, N. Spolaôr, C. S. R. Coy, F. C. Wu, “Web System Prototype based on speech recognition to construct medical reports in Brazilian Portuguese,” International Journal of Medical Informatics, vol. 121, pp. 39–52, 2019, doi: 10.1016/j.ijmedinf.2018.10.010.

L. F. Donnelly, R. Grzeszczuk, C. V. Guimaraes, W. Zhang, G. S. B. III, “Using a Natural Language Processing and Machine Learning Algorithm Program to Analyze InterRadiologist Report Style Variation and Compare Variation Between Radiologists When Using Highly Structured Versus More Free Text Reporting,” Current Problems in Diagnostic Radiology, vol. 48, no. 6, pp. 524–530, 2019, doi: 10.1067/j.cpradiol.2018.09.005.

H. Bay, T. Tuytelaars, L. Van Gool, “Surf: Speeded up robust features,” in Computer Vision – ECCV 2006, Berlin, Heidelberg, Germany, 2006, pp. 404–417, Springer Berlin Heidelberg.

E. Rublee, V. Rabaud, K. Konolige, G. Bradski, “Orb: An efficient alternative to sift or surf,” in Proceedings of the 2011 International Conference on Computer Vision, ICCV ’11, USA, 2011, p. 2564–2571, IEEE Computer Society.

P. F. Alcantarilla, A. Bartoli, A. J. Davison, “Kaze features,” in Proceedings of the 12th European Conference on Computer Vision - Volume Part VI, ECCV’12, Berlin, Heidelberg, 2012, p. 214–227, Springer-Verlag.

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Q. Weinberger Eds., Curran Associates, Inc., 2013, pp. 3111–3119.

J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 4171–4186, Association for Computational Linguistics.

M. Kusner, Y. Sun, N. Kolkin, K. Weinberger, “From word embeddings to document distances,” vol. 37 of Proceedings of Machine Learning Research, Lille, France, 07–09 Jul 2015, pp. 957–966, PMLR.

W. Boag, E. Sergeeva, S. Kulshreshtha, P. Szolovits, A. Rumshisky, T. Naumann, “Cliner 2.0: Accessible and accurate clinical concept extraction,” in ML4H: Machine Learning for Health Workshop at Advances in Neural Information Processing Systems, NIPS ’17, 2017.

J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, pp. 1234–1240, 09 2019, doi: 10.1093/bioinformatics/btz682.

I. Beltagy, K. Lo, A. Cohan, “SciBERT: A pretrained language model for scientific text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, Nov. 2019, pp. 3615–3620, Association for Computational Linguistics.

A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, S. Horng, “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific Data, vol. 6, p. 317, Dec 2019, doi: 10.1038/s41597-019-0322-0.

D. Demner-Fushman, S. Antani, M. Simpson, G. R. Thoma, “Design and development of a multimodal biomedical information retrieval system,” Journal of Computing Science and Engineering, vol. 6, no. 2, pp. 168–177, 2012, doi: 10.5626/JCSE.2012.6.2.168.

V. I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions and Reversals,” Soviet Physics Doklady, vol. 10, p. 707, Feb. 1966.