Interacting with Intelligent Digital Twins

This paper details the Human Computer Interaction (HCI) components of an Intelligent Digital Twin (IDT) system for a semiconductor manufacturing company using gamming visual effects, 3D Computer Aided Design (CAD) and sound effects. The designed digital twin (DT) system will allow users to detect any irregularities (such as equipment failures and defects) in the manufacturing processes, in a timely manner and test some changes without an actual touching of the physical components. The project aims to design an IDT system that enhances the user’s interaction by visualizing big data in such a way which can be easily understood and processed. Thus, allowing the user to intervene and control the entire production processes from the virtual console. The designed IDT system will enhance the UX of the system through the use of new interaction methodologies. The user will have full control of the data flow which is flowing from a data lake through for example, a gazing, a gesturing and a voice recognition interface which will provide contextual information based upon the user’s viewpoint. The current phase of the project investigates the use of Cave VR technology to improve the immersion and the interaction between the user and the virtual system. We seek to develop a virtual environment that makes the users feel they are naturally interacting in a visually-immersed environment irrespective of where they are located. By enhancing the interaction of the user with these new technologies, we will provide a better UX that would create an efficient system which reduces the overall costs of managing the plant and concretely realize the aspirations of Industry 4.0.