Large Language Models for in Situ Knowledge Documentation and Access With Augmented Reality
DOI:
https://doi.org/10.9781/ijimai.2023.09.002Keywords:
Augmented Reality, Deep Learning, Multimodal, Large Language Models, TransformerAbstract
Augmented reality (AR) has become a powerful tool for assisting operators in complex environments, such as shop floors, laboratories, and industrial settings. By displaying synthetic visual elements anchored in real environments and providing information for specific tasks, AR helps to improve efficiency and accuracy. However, a common bottleneck in these environments is introducing all necessary information, which often requires predefined structured formats and needs more ability for multimodal and Natural Language (NL) interaction. This work proposes a new method for dynamically documenting complex environments using AR in a multimodal, non-structured, and interactive manner. Our method employs Large Language Models (LLMs) to allow experts to describe elements from the real environment in NL and select corresponding AR elements in a dynamic and iterative process. This enables a more natural and flexible way of introducing information, allowing experts to describe the environment in their own words rather than being constrained by a predetermined structure. Any operator can then ask about any aspect of the environment in NL to receive a response and visual guidance from the AR system, thus allowing for a more natural and flexible way of introducing and retrieving information. These capabilities ultimately improve the effectiveness and efficiency of tasks in complex environments.
Downloads
References
H. Kagermann, J. Helbig, A. Hellinger, W. Wahlster, Recommendations for implementing the strategic initiative INDUSTRIE 4.0: Securing the future of German manufacturing industry; final report of the Industrie 4.0 Working Group. Forschungsunion, 2013.
R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, B. MacIntyre, “Recent advances in augmented reality,” IEEE Computer Graphics and Applications, vol. 21, no. 6, pp. 34–47, 2001, doi: 10.1109/38.963459.
M. Billinghurst, A. Clark, G. Lee, “A survey of augmented reality,” Foundations and Trends in Human- Computer Interaction, vol. 8, no. 2-3, pp. 73–272, 2015, doi: 10.1561/1100000049.
T. Caudell, D. Mizell, “Augmented reality: an application of heads-up display technology to manual manufacturing processes,” in Proceedings of the Twenty- Fifth Hawaii International Conference on System Sciences, 1992, pp. 659–669. doi: 10.1109/HICSS.1992.183317.
C. H. Chu, L. Wang, S. Liu, Y. Zhang, M. Menozzi, “Augmented reality in smart manufacturing: Enabling collaboration between humans and artificial intelligence,” Journal of Manufacturing Systems, vol. 61, pp. 658–659, 10 2021, doi: 10.1016/j.jmsy.2021.05.006.
S. Jaschke, “Mobile learning applications for technical vocational and engineering education: The use of competence snippets in laboratory courses and industry 4.0,” in Proceedings of 2014 International Conference on Interactive Collaborative Learning, ICL 2014, 1 2014, pp. 605–608, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/ICL.2014.7017840.
E. Marino, L. Barbieri, B. Colacino, A. K. Fleri, F. Bruno, “An Augmented Reality inspection tool to support workers in Industry 4.0 environments,” Computers in Industry, vol. 127, 5 2021, doi: 10.1016/j.compind.2021.103412.
M. Gattullo, G. W. Scurati, M. Fiorentino, A. E. Uva, F. Ferrise, M. Bordegoni, “Towards augmented reality manuals for industry 4.0: A methodology,” Robotics and Computer-Integrated Manufacturing, vol. 56, no. March 2018, pp. 276–286, 2019, doi: 10.1016/j.rcim.2018.10.001.
T. Masood, J. Egger, “Augmented reality in support of Industry 4.0—Implementation challenges and success factors,” Robotics and Computer-Integrated Manufacturing, vol. 58, pp. 181–195, 8 2019, doi: 10.1016/j.rcim.2019.02.003.
X. Xu, Y. Lu, B. Vogel-Heuser, L. Wang, “Industry 4.0 and Industry 5.0—Inception, conception and perception,” Journal of Manufacturing Systems, vol. 61, pp. 530–535, 10 2021, doi: 10.1016/j.jmsy.2021.10.006.
L. E. Garza, G. Pantoja, P. Ramírez, H. Ramírez, N. Rodríguez, E. González, R. Quintal, J. A. Pérez, “Augmented reality application for the maintenance of a flapper valve of a fuller-kynion type m pump,” Procedia Computer Science, vol. 25, pp. 154–160, 2013, doi: 10.1016/j.procs.2013.11.019.
J. Izquierdo-Domenech, J. Linares-Pellicer, J. Orta- Lopez, “Towards achieving a high degree of situational awareness and multimodal interaction with AR and semantic AI in industrial applications,” Multimedia Tools and Applications, vol. 82, pp. 15875–15901, 2023, doi: 10.1007/s11042-022- 13803-1.
J. Lave, E. Wenger, Situated learning: Legitimate peripheral participation. Cambridge university press, 1991. doi: 10.1017/CBO9780511815355.
C. A. Ventura, “Why switch from paper to electronic manuals?,” in Proceedings of the ACM conference on Document processing systems, 2000, pp. 111–116. doi: 10.1145/62506.62525.
F. Quint, F. Loch, “Using smart glasses to document maintenance processes,” Mensch und Computer 2015–Workshopband, pp. 203–208, 2015, doi: 10.1515/9783110443905-030.
C. Kollatsch, P. Klimant, “Efficient integration process of production data into Augmented Reality based maintenance of machine tools,” Production Engineering, vol. 15, pp. 311–319, 6 2021, doi: 10.1007/s11740-021-01026-6.
M. Hermann, T. Pentek, B. Otto, “Design principles for industrie 4.0 scenarios,” in Proceedings of the Annual Hawaii International Conference on System Sciences, vol. 2016-March, 3 2016, pp. 3928–3937, IEEE Computer Society. doi: 10.1109/HICSS.2016.488.
T. P. Carvalho, F. A. Soares, R. Vita, R. d. P. Francisco, J. P. Basto, S. G. Alcalá, “A systematic literature review of machine learning methods applied to predictive maintenance,” Computers and Industrial Engineering, vol. 137, 11 2019, doi: 10.1016/j.cie.2019.106024.
J. Ribeiro, R. Lima, T. Eckhardt, S. Paiva, “Robotic Process Automation and Artificial Intelligence in J. Sääski, T. Salonen, M. Hakkarainen, S. Siltanen, C. Woodward, J. Lempiäinen, “Integration of design and assembly using augmented reality,” in Micro- Assembly Technologies and Applications: IFIP TC5 WG5. 5 Fourth International Precision Assembly Seminar (IPAS’2008), Chamonix, France, 2008, pp. 295–404, Springer. doi: 10.1007/978-0-387-77405-3_39.
T. Salonen, J. Sääski, C. Woodward, O. Korkalo, I. Marstio, K. Rainio, “Data pipeline from CAD to AR based assembly instructions,” in Proceedings of the ASME/AFM World Conference on Innovative Virtual Reality 2009, WINVR2009, 2009, pp. 165–168. doi: 10.1115/WINVR2009-705.
M. Fiorentino, G. Monno, A. E. Uva, “Tangible digital master for product lifecycle management in augmented reality,” International Journal on Interactive Design and Manufacturing, vol. 3, no. 2, pp. 121–129, 2009, doi: 10.1007/s12008-009-0062-z.
M. Fiorentino, R. Radkowski, C. Stritzke, A. E. Uva, G. Monno, “Design review of CAD assemblies using bimanual natural interface,” International Journal on Interactive Design and Manufacturing, vol. 7, pp. 249– 260, 11 2013, doi: 10.1007/s12008-012-0179-3.
L. Hou, X. Wang, L. Bernold, P. E. D. Love, “Using Animated Augmented Reality to Cognitively Guide Assembly,” Journal of Computing in Civil Engineering, vol. 27, pp. 439–451, 9 2013, doi: 10.1061/(asce)cp.1943-5487.0000184.
L. Hou, X. Wang, M. Truijens, “Using Augmented Reality to Facilitate Piping Assembly: An Experiment-Based Evaluation,” Journal of Computing in Civil Engineering, vol. 29, 1 2015, doi: 10.1061/(ASCE)CP.1943-5487.0000344.
X. Wang, S. K. Ong, A. Y. Nee, “A comprehensive survey of augmented reality assembly research,” Advances in Manufacturing, vol. 4, pp. 1–22, 3 2016, doi: 10.1007/s40436-015-0131-4.
Industry 4.0 - A Literature review,” in Procedia Computer Science, vol. 181, 2021, pp. 51–58, Elsevier B.V. doi: 10.1016/j.procs.2021.01.104.
A. Bécue, I. Praça, J. Gama, “Artificial intelligence, cyber-threats and Industry 4.0: challenges and opportunities,” Artificial Intelligence Review, vol. 54, pp. 3849–3886, 6 2021, doi: 10.1007/s10462-020-09942-2.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, “Attention Is All You Need,” in Advances in neural information processing systems, vol. 30, 2017, pp. 6000– 6010. doi: 10.48550/arXiv.1706.03762.
G. W. Scurati, M. Gattullo, M. Fiorentino, F. Ferrise, M. Bordegoni, A. E. Uva, “Converting maintenance actions into standard symbols for Augmented Reality applications in Industry 4.0,” Computers in Industry, vol. 98, pp. 68–79, 6 2018, doi: 10.1016/j.compind.2018.02.001.
D. K. Baroroh, C. H. Chu, L. Wang, “Systematic literature review on augmented reality in smart manufacturing: Collaboration between human and computational intelligence,” Journal of Manufacturing Systems, vol. 61, pp. 696–711, 10 2021, doi: 10.1016/j.jmsy.2020.10.017.
P. Wang, X. Bai, M. Billinghurst, S. Zhang, X. Zhang, S. Wang, W. He, Y. Yan, H. Ji, “AR/MR Remote Collaboration on Physical Tasks: A Review,” Robotics and Computer-Integrated Manufacturing, vol. 72, 12 2021, doi: 10.1016/j.rcim.2020.102071.
M. Sereno, X. Wang, L. Besancon, M. J. McGuffin, T. Isenberg, “Collaborative Work in Augmented Reality: A Survey,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, pp. 2530–2549, 6 2022, doi: 10.1109/TVCG.2020.3032761.
B. Marques, S. Silva, J. Alves, T. Araujo, P. Dias, B. S. Santos, “A Conceptual Model and Taxonomy for Collaborative Augmented Reality,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, pp. 5113–5133, 12 2022, doi: 10.1109/TVCG.2021.3101545.
B. Marques, S. Silva, J. Alves, A. Rocha, P. Dias, B. S. Santos, “Remote collaboration in maintenance contexts using augmented reality: insights from a participatory process,” International Journal on Interactive Design and Manufacturing, vol. 16, pp. 419–438, 3 2022, doi: 10.1007/s12008-021-00798-6.
V. Elia, M. G. Gnoni, A. Lanzilotto, “Evaluating the application of augmented reality devices in manufacturing from a process point of view: An AHP based model,” Expert Systems with Applications, vol. 63, pp. 187–197, 11 2016, doi: 10.1016/j.eswa.2016.07.006.
M. L. Yuan, S. K. Ong, A. Y. Nee, “Augmented reality for assembly guidance using a virtual interactive tool,” International Journal of Production Research, vol. 46, pp. 1745–1767, 4 2008, doi: 10.1080/00207540600972935.
S. K. Ong, Z. B. Wang, “Augmented assembly technologies based on 3D bare-hand interaction,” CIRP Annals - Manufacturing Technology, vol. 60, no. 1, pp. 1– 4, 2011, doi: 10.1016/j.cirp.2011.03.001.
D. Mourtzis, V. Siatras, J. Angelopoulos, “Real-time remote maintenance support based on augmented reality (AR),” Applied Sciences (Switzerland), vol. 10, 3 2020, doi: 10.3390/app10051855.
A. Gilchrist, “Introducing Industry 4.0,” in Industry 4.0, Springer, 2016, ch. 13, pp. 195–215, doi: 10.1007/978-1-4842-2047-4.
Z. Ziaei, A. Hahto, J. Mattila, M. Siuko, L. Semeraro, “Real-time markerless Augmented Reality for Remote Handling system in bad viewing conditions,” Fusion Engineering and Design, vol. 86, pp. 2033–2038, 10 2011, doi: 10.1016/j.fusengdes.2010.12.082.
D. Tatić, B. Tešić, “The application of augmented reality technologies for the improvement of occupational safety in an industrial environment,” Computers in Industry, vol. 85, pp. 1–10, 2 2017, doi: 10.1016/j.compind.2016.11.004.
A. Syberfeldt, O. Danielsson, M. Holm, L. Wang, “Dynamic Operator Instructions Based on Augmented Reality and Rule-based Expert Systems,” Procedia CIRP, vol. 41, pp. 346–351, 2016, doi: 10.1016/j.procir.2015.12.113.
R. Palmarini, I. Fernández, D. Amo, D. Ariansyah, S. Khan, J. A. Erkoyuncu, R. Roy, “Fast Augmented Reality Authoring: Fast Creation of AR step-by-step Procedures for Maintenance Operations,” IEEE Access, vol. 11, pp. 8407-8421, 2023, doi: 10.1109/ACCESS.2023.3235871.
T. Zonta, C. A. da Costa, R. da Rosa Righi, M. J. de Lima, E. S. da Trindade, G. P. Li, “Predictive maintenance in the Industry 4.0: A systematic literature review,” Computers and Industrial Engineering, vol. 150, 12 2020, doi: 10.1016/j.cie.2020.106889.
M. Casillo, F. Colace, L. Fabbri, M. Lombardi, A. Romano, D. Santaniello, “Chatbot in industry 4.0: An approach for training new employees,” in Proceedings of 2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering, TALE 2020, 12 2020, pp. 371–376, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/TALE48869.2020.9368339.
N. U. Moroff, E. Kurt, J. Kamphues, “Machine Learning and Statistics: A Study for assessing innovative Demand Forecasting Models,” Procedia Computer Science, vol. 180, pp. 40–49, 2021, doi: 10.1016/j.procs.2021.01.127.
B. Maschler, M. Weyrich, “Deep Transfer Learning for Industrial Automation: A Review and Discussion of New Techniques for Data-Driven Machine Learning,” IEEE Industrial Electronics Magazine, vol. 15, pp. 65–75, 6 2021, doi: 10.1109/MIE.2020.3034884.
J. Redmon, A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, 2018, doi: 10.48550/arXiv.1804.02767.
S. Chidambaram, H. Huang, F. He, X. Qian, A. M. Villanueva, T. S. Redick, W. Stuerzlinger, K. Ramani, “ProcessAR: An augmented reality-based tool to create in-situ procedural 2D/3D AR Instructions,” in DIS 2021 - Proceedings of the 2021 ACM Designing Interactive Systems Conference: Nowhere and Everywhere, 6 2021, pp. 234–249, Association for Computing Machinery, Inc. doi: 10.1145/3461778.3462126.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, “On the Opportunities and Risks of Foundation Models,” arXiv preprint arXiv:2108.07258, 8 2021, doi: 10.48550/arXiv.2108.07258.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10684–10695. doi: 10.48550/arXiv.2112.10752.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018.
J. M. Rožanec, I. Novalija, P. Zajec, K. Kenda, H. Tavakoli Ghinani, S. Suh, E. Veliou, D. Papamartzivanos, T. Giannetsos, S. A. Menesidou, R. Alonso, N. Cauli, A. Meloni, D. R. Recupero, D. Kyriazis, G. Sofianidis, S. Theodoropoulos, B. Fortuna, D. Mladenić, J. Soldatos, “Human- centric artificial intelligence architecture for industry 5.0 applications,” International Journal of Production Research, vol. 61, no. 20, pp. 6847–6872, 2022, doi: 10.1080/00207543.2022.2138611.
A. Akundi, D. Euresti, S. Luna, W. Ankobiah, A. Lopes, I. Edinbarough, “State of Industry 5.0—Analysis and Identification of Current Research Trends,” Applied System Innovation, vol. 5, 2 2022, doi: 10.3390/asi5010027.
O. Hardt, K. Nader, L. Nadel, “Decay happens: The role of active forgetting in memory,” Trends in Cognitive Sciences, vol. 17, pp. 111–120, 3 2013, doi: 10.1016/j.tics.2013.01.001.
O. O. Adesope, D. A. Trevisan, N. Sundararajan, “Rethinking the Use of Tests: A Meta-Analysis of Practice Testing,” Review of Educational Research, vol. 87, pp. 659–701, 6 2017, doi: 10.3102/0034654316689306.
J. D. Karpicke, H. L. Roediger, “The critical importance of retrieval for learning,” Science, vol. 319, pp. 966–968, 2 2008, doi: 10.1126/science.1152408.
D. Bissig, C. Lustig, “Who benefits from memory training?,” Psychological Science, vol. 18, pp. 720–726, 8 2007, doi: 10.1111/j.1467-9280.2007.01966.x.
M. Wolf, M. Kleindienst, C. Ramsauer, C. Zierler, E. Winter, “Current and future industrial challenges: demographic change and measures for elderly workers in industry 4.0,” Annals of the Faculty of Engineering Hunedoara, vol. 16, no. 1, pp. 67–76, 2018.
N. J. Cepeda, H. Pashler, E. Vul, J. T. Wixted, D. Rohrer, “Distributed practice in verbal recall tasks: A review and quantitative synthesis,” Psychological Bulletin, vol. 132, pp. 354–380, 5 2006, doi: 10.1037/0033- 2909.132.3.354.
S. K. Carpenter, N. J. Cepeda, D. Rohrer, S. H. Kang, H. Pashler, “Using Spacing to Enhance Diverse Forms of Learning: Review of Recent Research and Implications for Instruction,” Educational Psychology Review, vol. 24, pp. 369–378, 9 2012, doi: 10.1007/s10648-012-9205-z.
Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, Y. Goldberg, “Measuring and Improving Consistency in Pretrained Language Models,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 1012–1031, 12 2021, doi: 10.1162/tacl_a_00410.
V. Sanh, L. Debut, J. Chaumond, T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 10 2019, doi: 10.48550/arXiv.1910.01108.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv preprint arXiv:1907.11692, 7 2019, doi: 10.48550/arXiv.1907.11692.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” The Journal of Machine Learning Research, vol. 21, pp. 5485–5551, 10 2020, doi: 10.48550/arXiv.1910.10683.
T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,” arXiv preprint arXiv:2211.05100, 11 2022, doi: 10.48550/arXiv.2211.05100.
S. Arroni, Y. Galán, X. Guzmán-Guzmán, E. R. Nuñez-Valdez, A. Gómez, “Sentiment Analysis and Classification of Hotel Opinions in Twitter With the Transformer Architecture,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 8, no. 1, p. 53-63, 2023, doi: 10.9781/ijimai.2023.02.005.
Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, H. S. Zheng, D. Zhou, N. Houlsby, D. Metzler, “UL2: Unifying Language Learning Paradigms,” arXiv preprint arXiv:2205.05131, 5 2022, doi: 10.48550/arXiv.2205.05131.
Y. Tay, J. Wei, H. W. Chung, V. Q. Tran, D. R. So, S. Shakeri, X. Garcia, H. S. Zheng, J. Rao, Chowdhery, D. Zhou, D. Metzler, S. Petrov, N. Houlsby, Q. V. Le, M. Dehghani, “Transcending Scaling Laws with 0.1% Extra Compute,” arXiv preprint arXiv:2210.11399, 10 2022, doi: 10.48550/arXiv.2210.11399.
Together, “GPT-JT,” 2022. [Online]. Available: https://huggingface.co/togethercomputer/GPT-JT-6B-v1.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” arXiv preprint arXiv:2212.04356, 12 2022.
Downloads
Published
-
Abstract221
-
PDF76
Submit Article
Information
