Revitalizing Human-Robot Interaction: Phygital Twin Driven Robot Avatar for China–Sweden Teleoperation

Digital Twin (DT) is built and maintained in the digital rather physical realm [1, 2]. Dynamic DT mirrors the exact state of the physical system by means of multidimensional models describing the behavior of physical entity, and sensors providing the real-time coupling to models [3]. The outbreak of COVID-19 pandemic results in speeding up the virtualization and digitization of the physical entity with the support from DT. Nevertheless, quarantine and division led by COVID-19 pandemic make human beings’ eyes for the physical interaction become more eager. The physical world and the digital world are becoming interwoven, which indicates a new opportunity towards phygital twin. As is so often the question: how can I get closer to my distant companion? How can I extend my service or emotional concern to facilitate interaction in spite of space obstruction? These questions deserve thinking as a lesson learned from outbreak of COVID-19 pandemic.

DT focuses on how digital elements to represent physical objects in the abstract and condensed ways based on a confluence of technological advances [4]. Intrinsic difference between human beings and physical systems leads to different emphasis in the description by DT [5]. When involved into human-centric DT, human users expect tangible and intuitive experience to meet requirements of engagement and immersion. They need to prioritize transparency and intuition providing for humans. Moreover, not only interaction enhancement, but also getting rid of the limitations of space are primary requirements of the next-generation DT in favor of human’s involvement. Phygital twin emerges at this critical moment.

“Phygital” is a portmanteau of “physical” and “digital”. It is the concept of utilizing technology to bridge the reality and cross-space reality by means of interaction between digital world and physical world, aiming for enhancing unique and interactive experiences of users, providing a positive outcome with a seamless toggling between the physical realm and the digital realm.

Phygital twin is a term allowing trio among physical reality, digital world, and the other cross-space physical reality. Its connection changes the paradigm by unifying multiple hardware infrastructure and software infrastructure [6]. With advanced technology, phygital twin has an opportunity to endow avatar with more rational and emotional factors. Imaging taking care of distant family, sharing your joy or sorrows with distant friend, receiving warm hug from lovers, surpass verbal communication [7].

In terms of realization, robot is quite an appropriate vehicle of phygital twin thanks to its advanced technologies and broad application [8, 9]. Telerobotics is one of the earliest aspects of robotics. It refers to robotics with a human operator in control or human-in-the-loop. Furthermore, human is responsible for any high-level planning, or cognitive decisions while robot executes mechanical implementation. Researchers attempt to apply human intelligence, cognition, and skills to robots. Selvaggio et al. introduced the first haptic-guided shared-control teleoperation architecture [10]. It is used in the transportation of an object on a tray-like end-effector. It allows integrating human operators’ experience and cognitive capabilities with the precision, reactivity, and repeatability of autonomous systems. In addition to the control strategy, the transparency and stability are also challenging. Feizi et al. proposed a real-time frequency-based delay compensation approach to maximize transparency while reducing the activation of the stabilization layer [11]. However, human-robot teleoperation is the epitome of phygital twin in terms of technology scale while phygital twin does not just refer to operation of systems or machines at a distance. Zhang et al presented a mixed-reality avatar attached with robot in the dynamic interaction among humans, robot, and mixed-reality avatar robot [12]. Beik-Mohammadi et al. integrated reinforcement learning with the model mediated teleoperation. It can handle challenging problems in a long-distance teleoperated grasping scenario under long time-delays [13]. In comparison with telerobotics, researchers related to phygital twin strive to create an environment for maximizing the value of time, transcending space, and extending humans’ ability. No matter how far away you are, where you are, and who you are, robot becomes your avatar to do what you want to do without needing to be there in person. Meanwhile, real-world experiences and virtual world experiences are cherished for humans. Thus, robot avatar is expected to perceive real-world, thereby facilitate human beings to sense real world and virtual world without constraints of place.

In this work, researchers step further on transcontinental phygital twin system. Section 2 introduces the system architecture of the proposed system, and Section 3 shows validation case study. Conclusion is given in Section 4.

Phygital twin focuses on applying knowledge and abilities gained from human beings to phygital avatar, i.e., transferring information from humans with subjective initiative to physical avatar (e.g., robot). The integrated phygital twin system involves the operator located in Sweden, and the robot avatar in China (Figure 1). A set of wearable devices is used to capture human’s motion and provide feedback information. The robot receives transmitted information, replacing human for executing tasks in a first-person perspective.

The phygital twin goes after releasing potential abilities of the avatar system and combining strengths of the operator and the robot together. Inspired by robot avatar, the phygital twin system is validated in a transcontinental demonstration involving the human operator and the robot system. Human wearing the InMoCap located in Västerås, Sweden, controls the robot system located in Hangzhou, China.

Effective and reliable human motion and robotic motion lay the foundation for collaborative interaction. Characteristics of InMoCap are shown in Table 1. The computed human motion data have been measured using the optical system as golden standard. The Root Mean Square Error (RMSE) of joint angles in the static poses is 3.9° while is 4.7° in the dynamic movements. The Pearson correlation coefficient of dynamic motion data is about 92.5%, and the average agreement certainty of the overall joint angles is about 97.8%. RMSE between IMU-based computed motion trajectories and standard values is about 1.2 cm.

With the mapped motion trajectories from human operator, robotic arms replace human arms in the operation tasks, for example, placing blocks into container (Figure 3(a). The active robotic arm takes on the primary part of task, i.e., picking up the target and placing it down. By moving container, the passive robotic arm collaborates with the active arm in the placement of the target. Furthermore, dual-arm operation ability shows humans’ strength in interaction. Robot system approaches targeted objects and executes task with the collaboration of two robotic arms. As shown in Figure 3(b), robot avatar fetches bamboo slip on the desk, and the other robotic arm fetches bamboo tube. Human operator needs to coordinate two arms to put the bamboo slip into the tube. In the operation tasks, basic feedback devices are equipped with operators. For example, when operator teleoperates the robot, robotic perception information is transmitted to operator. Specifically, operator needs to know if the robotic grippers open or not according to results of grabbing tasks. With the acquired information from vibration and flash, direct feedback information helps operator make decision conveniently.

This paper presents preliminary reflections and discussions on phygital twin as a novel human-robot interaction paradigm, including preliminary framework, components, and validation case. Existing human-machine interaction paradigm is at the transitional stage where multiple enabling technologies are provided for extension and enrichment of related conceptual framework. In this paper, phygital twin facilitates interaction between human and robot avatar. Thereinto, robot avatar replaces human to execute multiple tasks with providing more sense of perception, feedback, and cognition. In particular, operation cases have been completed using transcontinental teleoperation. With increasing awareness of bridging the physical reality and cross-space physical reality, it is foreseeable that phygital twin will overthrow traditional ideas and join in next-generation human-machine paradigm.

Challenges remain in realizing and promoting phygital twin system. To promote and popularize phygital twin, it is critical to clarify a comprehensive and basic framework for implementing phygital twin in varying industrial levels. Despite all the optimism around phygital twin, it is known that we are at the early stage of adopting this vision in multiple areas. It is expected to embrace the future development across the broader application areas, such as robotics, cognitive computing, and artificial intelligence. However, there are many challenges remaining to solve, such as the availability of data from which to build phygital twin system, safety issues related to the integration of robot avatar into human-centric system. Moreover, the development of phygital twin also requires the legal and political support. The growing awareness of the vision of phygital twin will certainly bring more discussions about handling with challenges and promoting phygital twin.