Automated Information Supply of Worker Guidance Systems in Smart Assembly Environment

Digital assistance systems (DAS) are used within a CPAS as interface between humans and technical systems [10]. The primary goal is to provide optimal worker support to increase productivity, reduce execution times, minimize error rates and enable end-to-end traceability [11]. DAS comprise basic functions including documentation of process data, monitoring of processes, decision support and information output [12]. For information output, the term “worker information systems” (WIS) is used in literature of production management. WIS provide information such as step-by-step assembly instructions, security hints or warnings of potential errors without the need of printed paper media [13].

For step-by-step guidance of workers through assembly processes, also the term worker guidance, respectively WGS is used. WGS allow workers to overcome difficulties in performing complex precision assembly processes and reduce cognitive burden in assembling small lot sizes of ever increasing product variants [14]. The most significant difference between WIS and WGS is the feedback loop: WIS only supply information assistance according to a given set of rules, while WGS additionally support the input of information and data manually through graphical user interfaces or automatically through different sensors [15]. Aehnelt et al. stated that “information assistance in form of guiding can be understood as an informal way of mediating and learning facts (what), procedures (how) and concepts (why) required for a specific assembly task”. Therefore a worker has to remember, understand and apply the information to execute the assembly task [16].

Lušić et al. differ between static and dynamic provision of information as well as real versus virtual information. Text and pictures are time-invariant and therefore static information, leading to additional cognitive load of the worker. Dynamic provision of information, e.g. videos or 3D animations lead to less cognitive load, but the duration of these have to be adapted to individual worker’s needs. Real information require real objects for their creation and include recorded photos or videos, while virtual information can be derived digitally e.g. using a 3D Computer-Aided Design (CAD) software [17].

The information provided by WGS can be in form of texts, pictures, videos, virtual 3D objects or simple light signals and must be prepared, programmed and transferred to databases or storage media of an individual target system prior to production [18]. This preparation process is very time consuming and usually requires a specialized knowledge in programming and 3D CAD modelling [19]. In case of a small or single lot size production, the described preparation process has to be carried out often and represents a significant cost factor, which furthermore prevents an efficient usage of WGS [20]. To cope with the aforementioned challenges, different approaches have been presented in recent studies, which can be clustered into following categories:

(i) automation of assembly sequence planning [21], (ii) automation of instruction information creation [22], (iii) automated entry of created information into target systems [23] and (iv) support the human assembly planner where automation is not possible [24]. These four categories are described in detail:

(i) Automation of assembly sequence planning: Since the early 1990s, various algorithms and heuristics have been developed to automatically derive feasible assembly sequences of a product variant from product data or 3D CAD models, e.g. [25, 26]. This research area evolved with more computing power: The original approaches considering a simple listing of assembly sequences were developed successively, so that modern solutions allow an automatic feasibility study with regard to stability and available space at the joining position of each part [27], but also the average required assembly time can be calculated [28]. All of the aforementioned approaches relate to the general assembly planning process, but are not designed to create, process or distribute information for WGS.

(ii) Automation of instruction information creation: Mader et al. describe an approach to be able to automatically create work instructions in textual form and as pictures based on geometry and workstation data [22]. More recent work describes the preparation of videos and assembly animations using virtual prototypes [13]. Sääski et al. describe a concept to automatically create 3D objects for Augmented Reality (AR) worker guidance. Hereby the focus has been set on the integration of a wide variety of information systems as consistently as possible [29]. The created information has to be entered manually into databases of target WGS using a graphical user interface (GUI). This step is also associated with high manual workload during preparation phase.

To ensure that assembly workers on shop floor can use the created instruction information, it must be entered into the database of WGS through software interfaces. Müller et al. describe an exchange of information between agents and modules. While a WGS can be seen as a module, an “agent acts as a mediator or coordinator” between these modules and the virtual assembly planning environment [23]. A similar approach is pursued by Fischer et al., who describe the data flow between virtual assembly planning and the WIS database. Data is exported from the planning environment, translated into the desired target structure via an associative array and can be imported into the WGS [30].

(iv) Support the human assembly planner where automation is not possible: Zauner et al. describe the use of domain specific wizards, so-called “authoring wizards” in order to create visual information in a user-friendly way and without any programming knowledge [31]. Through a GUI, an assembly planner defines the required assembly information, such as assembly sequence, parameters and required tools [32]. The described approach is widely used in context of AR solutions and is applied in research and industry [33]. Despite support by means of authoring software, high manual effort remains in creation and entry of the information for each product variant. In addition, these software packages are usually limited to AR worker guidance and are designed for specific output devices or an individual WGS solution. Sensors for detecting depth information and movements enable teach-in of work instruction content at the assembly stations directly [34]. Funk et al. have developed a projection based WGS, which allows a complex assembly process to be trained by experienced workers. During the assembly process, the system recognizes the required part containers as well as joining positions and derives all the information required for projection-based worker guidance automatically. However, the authors themselves point out that this system is not mature and further development must be made, e.g. optimization of workpiece detection [20]. In addition, such a system cannot be used in lot size 1 production, since the entire assembly process has to be taught in with at least one piece.

In summary, the state of the art includes partial solutions, which favour a reduction of expenses in the supply of instructional information e.g. through automation of preparation tasks or support of human assembly planners. However, the lack of a holistic and consistent approach in order to achieve a fully configured WGS even at complex products and small lot size is evident. In this paper, we present an approach for the automated information supply of worker guidance systems, which helps to significantly reduce content creation efforts and to relieve assembly planning staff, especially in smart assembly environments. The approach differs from the state of the art by a continuous processing chain from product development to the output of digital content information on assembly shop floor. The activities of information supply of worker guidance systems are divided between human and computer according to their respective strengths and weaknesses. While human planners contribute product-, process- and resource-knowledge by means of optimally designed input interfaces, a computer takes over time-consuming creation activities for instruction elements, e.g. texts, pictures or optimised 3D models for AR. In order to further relieve assembly planners, they are supported by machine learning at the time-consuming task of planning assembly sequences. Case-based reasoning is used to derive assembly sequences for the new product variant based on earlier planning knowledge of similar product variants automatically. The following sections describe a conceptual design for an automated information supply and the technical implementation in a test environment.

In the context of WGS, the authors propose a definition for the term “information supply” as combination of “information creation” and “information entry”. Hereby, the task of information creation contains the following subtasks:

In order to provide instructions to workers on the shop floor, the instructional information has to be transferred into a database or file system of the target WGS. Most WGS provide a backend editor or a similar GUI, which can be used by assembly planners in order to convert created information to the required format and to enter it into the database. The information supply process should be carried out for each individual product variant and thus leads to high manual effort and costs for small lot sizes. This paper proposes a holistic knowledge-based approach, which includes entire information supply process, taking over routine tasks through algorithms and supporting assembly planners with a self-learning expert system. The result is a division of tasks between human and algorithm. While humans provide their domain and process knowledge, algorithms take repetitive tasks such as derivation of virtual information as well as transfer of created instruction information to databases.

Figure 1 illustrates the approach and describes how data is processed so that a 3D CAD model of a product variant can be used to adequately supply information to a WGS. The blue boxes symbolise automated algorithms, while green boxes designate GUIs for interaction with human planners. The approach builds upon earlier developments by Reisinger et al. [35] and has been extended by additional concepts to further reduce manual effort.

In first step “Authoring Process”, human assembly planners define assembly sequence (“what has to be done”) and work methods (“how is it done”). An authoring tool with 3D user interface is provided and visualizes the 3D CAD model of current product variant. By selecting parts and subassemblies, assembly steps as well as an assembly sequence are defined. Individual steps can be enriched with additional information, e.g. required tools, screwing torques or parts. Furthermore, the assembly planner specifies intermediate steps that cannot be automatically extracted from a 3D CAD model, e.g. missing parts like springs or pipes. During this authoring process, the assembly planner is assisted through various functions, e.g. suggestion of correct torque depending on dimension of the screw. In summary, the main purpose of the authoring process is to enter process- and resource-knowledge of human assembly planners in an efficient way. Previous work in literature describing similar authoring tools – e.g. [31, 36] or [24] – show a significant weakness: The authoring process has to be conducted for each product variant, even in case of similar 3D product models. This leads to a high manual effort for assembly planners and thus a low efficiency is resulting.

To deal with the described weakness of conventional authoring processes, additional system features in planning new product variants are considered:

Creating virtual instruction information for individual assembly steps is conducted automatically and requires the following domain knowledge elements:

The textual description of individual assembly steps is automatically derived using previously manually defined text modules and contains information about assembled components, the operation and needed tools. Defined colour codes enable a differentiation of small and big parts, standard parts as well as a differentiation of tools.

The pictures for worker guidance are automatically derived as CAD screenshots for each assembly step and relevant components are highlighted in colour. To ensure optimal visibility of every component, pictures are derived from two different perspectives: A fixed position and viewing angle, in order to provide overall orientation to the assembly worker, as well as a picture from a view perspective previously defined in the “Authoring Process”. To guarantee a high standardization of instructions, the colour coding of the pictures is the same that is used for the textual descriptions.

The required information for worker guidance has to be prepared and entered into a database or file system of the target WGS so that worker guidance information on shop floor can be used to support assembly workers. The data structure of the database and file system is defined differently for each WGS and no standardized interface for entering worker information exists. At this point, the principle of post-processing is taken from Computer-Aided Manufacturing (CAM) information chain [39] and we propose the following definition analogously for WGS: A post-processor in context of worker guidance systems is an algorithm for automated translation of prepared worker information (assembly sequence, texts, pictures) and entry of these information into a database or file system of a target WGS. A post-processor is developed once for each target system and has to be adapted in regard to new data structures in case of updates. The development requires direct writing access to the respective database or file system as well as a good knowledge of data structure of the WGS.

During implementation, individual modules have been developed using suitable programming languages and existing frameworks, e.g. authoring tool, generator for texts and pictures as well as post-processors. In order to ensure an efficient planning process, attention was paid to easy maintainability of the data structures as well as a high user-friendliness of the GUI.

Figure 2 presents the GUI of the developed authoring tool, which is used by the assembly planner to enter process- and resource-related information. The open source software “FreeCAD” is used to provide a 3D environment. FreeCAD and authoring window together form the authoring tool. Through python scripts as well as a provided Application Programming Interface (API), FreeCAD is able to communicate with the developed authoring window, which is displayed at the right side of above screenshot. The parts and components of the product variant are selected by the assembly planner for each assembly step through a list or by directly clicking parts and components in 3D environment. Screws are automatically identified by part name and appropriate screwing parameters such as torque, depth and pitch are suggested.

Once the authoring process is completed and the information is released, it is passed to a virtual machine connected to local area network (LAN) via a Representational State Transfer (REST) API. The virtual machine acts as a container for various algorithms running automatically as well as a SQL database for data storage. The worker information is automatically processed as described in previous chapter. The automatic generation of instruction texts is carried out by algorithms, which have been implemented in Java Programming Language. The creation of instruction pictures is done by a self-developed media generator. Because of high resource usage, the media generator is implemented as a cloud service and can be controlled via a REST API. The assembly plan in XML form (created by authoring tool) and a 3D CAD model of the product variant to be assembled serve as input for the media generator. The output are generated pictures from different perspectives, which are automatically downloaded after the creation process is completed.

The developed software-based PoCD has been integrated into PFI40 and linked with existing information systems and WGS. The PFI40 is located in Vienna (Austria) and “serves both as a research platform and a teaching and training environment with regard to a human-centered cyber-physical production system” for lot size 1 production [40]. It combines development and testing of prototypes (Pilot Factory), demonstration and communication of findings (Demonstration Factory) and transfer of knowledge to students and course participants from practice (Learning Factory). The production of a plastic 3D printer is demonstrated in a realistic environment of approximately 900 m² space. Industrial machines, 3D printers, logistics systems and a cyber-physical assembly line are available [41]. The assembly line of PFI40 consists of four cyber-physical assembly stations, which in turn consist of various assistance systems, including visual worker guidance, an intelligent screwing system as well as collaborative robots.

Figure 3 shows the integration of the software based PoCD in the IT landscape of PFI40 [42]. Red coloured elements represent already existing systems of the PFI40 and contain an ERP system, a 3D CAD environment as well as two WGS of different manufacturers. The green ellipse indicates the virtual machine and contains a central database of the PoCD, implemented algorithms as well as passive and active interfaces for communication with the system environment. The authoring window is not part of the virtual machine but is executed on desktop computer of the assembly planner and communicates with 3D CAD environment FreeCAD and via a REST API with the virtual machine. After successful planning and preparation of the worker information, it is transferred towards two target WGS “Armbruster ELAM” [43] and “Sarissa QA” [44] using developed post-processors. Precision assembly staff on shop floor is now able to use these instructions for step-by-step worker guidance.

This paper presents a novel knowledge-based approach to reduce the required manual effort in information supply process by dividing preparation tasks between human and machine. This involves the use of pre-defined rule-based algorithms for generating instructions, self-learning expert systems for transferring assembly plans to similar product variants and agent-based interfaces (post-processors) for automated entry of created information. The development of the proposed post-processors requires intensive knowledge of the data structure of the target WGS. Therefore, we recommend that the manufacturer of WGS should do the development of the post-processors.

The approach has been implemented as a software-based PoCD, integrated into IT environment of PFI40 and is used for information supply of two WGS of a cyber-physical assembly line. Previous findings show that preparation efforts per product variant could be reduced significantly and the system works efficiently even if there are computationally intensive algorithms. A comprehensive user study with both experienced and inexperienced participants is planned in order to measure actual effects, such as a reduction in planning times or an increase in worker satisfaction.

The approach presented in this paper is limited to providing information to WGS for assembly activities. Due to a recognizable evolution of conventional assembly systems into cyber-physical assembly systems, we recommend further research in the field of automated information supply of these environments. In addition to WGS, this also includes creation and linking of configuration sets for devices, tools, machines and measuring equipment. A further starting point for development is the expansion of the field of application so that, in addition to assembly activities, WGS can also be automatically supplied with information for manual maintenance, set-up and servicing processes. The approach presented here can also be used for mutual (reciprocal) learning between human and machine. For example, a machine (e.g. algorithm) can learn from human (e.g. process planner) how to plan precision assembly processes and how to create associated worker instructions for WGS. Furthermore, a precision assembly worker can also learn from the machine by using the instructions of the WGS [45]. A transfer and adaptation of the presented approach towards the creation of CNC machine tool code is also worth researching and would make work preparation of cutting machine tools more efficient.