Design of a self-learning multi-agent framework for the adaptation of modular production systems

Different manufacturing paradigms, together with physical and logical enablers, have been introduced to facilitate system changes [9]. The paradigms include flexible and reconfigurable manufacturing [10, 11], holonic and agent-based manufacturing (see, for example, the ADACOR architecture by [12]) and evolvable assembly systems [13]. A recent review of reconfigurable manufacturing systems is conducted by [14]. Reviews of agent-based manufacturing are presented by [15] and [16]. The contributions of holonic manufacturing to Industry 4.0 are examined by [17]. Although agent-based and holonic manufacturing provide clear benefits, in particular in terms of flexibility and robustness, there are still barriers to their widespread industrial adoption. The fact that there is no guarantee on the operational performance of agent-based solutions prevents them from being applied to real-time control problems and is an obstacle to their acceptance by the management of companies.

System adaptations can be made to increase production, improve a production process, or produce a product variant. They can be either physical or logical, each having different objectives. Physical adaptations are aimed at increasing production capacity or producing a product variant, and include operations such as the addition or modification of modules or machines. Logical adaptations are aimed at improving the utilisation of available resources, do not require hardware alterations and involve changing parameters or reprogramming to implement, for instance, a different scheduling or routing policy.

In particular, logical adaptations can be determined and applied autonomously by intelligent agents. Multi-agent systems exhibit many self-* capabilities and self-organising multi-agent mechatronic systems have been developed in successful academic and industrial projects [18], with typical applications including manufacturing execution systems [19, 20], routing and scheduling [21]. Self-organising behaviour is also characteristic of some natural systems, which influenced the development of new computational techniques. A recent application of multi-agent systems is, for example, distributed diagnosis with bio-inspired algorithms [22]. A holonic manufacturing execution system with control mechanisms inspired by natural systems is presented by [23]. The integration of a centralised scheduling system based on constraint programming with a holonic manufacturing execution system is described by [24]. Hybrid solutions like this one are worthwhile investigating because they benefit from the strengths of both approaches. The holonic systems of [23] and [24], along with many others, implement the popular reference architecture PROSA (see [25] for a historical perspective). It would be interesting to compare different architectures or even different implementations of the same reference architecture, defining criteria for their evaluation. Also, there is a need for agent-based design patterns, similarly to object-oriented design patterns.

Another concept related to distributed intelligence is product-driven automation. Two heterarchical architectures for intelligent products are presented by [26]. Product intelligence seeks to give customers greater control over the processing of orders [27].

Agent-based manufacturing systems have several distinctive characteristics in common with cyber-physical systems (CPS), as they are both intelligent and distributed systems offering high levels of adaptability through the cooperation of interconnected entities. As a matter of fact, a multi-agent system can be used to implement the cyber part of a CPS. A survey of the adoption of agents in industrial CPS is conducted by [18].

According to the US National Science Foundation, CPS are “engineered systems that are built from, and depend upon, the seamless integration of computation and physical components”. Although systems combining physical and computational elements have long been in existence, the design of one type of elements has normally met only some minimum interface requirements and not taken full advantage of the capabilities of the other type. To integrate cyber and physical systems, the approaches of cyberizing the physical and physicalizing the cyber are identified by [28]. Some research work in this direction leverages the multi-agent paradigm. The integrated development of multi-agent PLC-based control systems using IEC 61131-3 is discussed by [29]. The combination of mechatronic production systems and multi-agent systems is investigated by [30]. What is missing in many works is a general methodology for designing agent-based control systems, which would facilitate new developments and applications. A model-based methodology (DACS) to enable an ordinary engineer to design an agent-based control system is described by [31]. More research in this direction is required to promote the adoption of multi-agent systems in CPS.

Figure 1 gives an overview of the framework architecture. The various components and their relationships will be explained in the next subsections. The following types of agents are identified: Multiple instances of an agent type can be created and deployed on different physical components or layers, as discussed in Section 3.5.

Experience recognition (ER) is the process of recording the adjustments being made to the machine and evaluating their impact. An adaptation usually involves a number of adjustments. By adjustment we mean an atomic change that cannot be broken down into separate changes. Examples of adjustments are changing pick-and-place points, cam angles, speed, pressure, grippers or pallet geometry, or even using different part styles. Some adjustments can be performed through software (e.g. pick-and-place points), while others require physical operations. In the former case, the adjustments are captured by the framework automatically. In the latter case, they must be entered manually. The effect of an adjustment must be measurable by the KPI, in the sense that a difference in performance should be reflected in the KPI value. If this is not the case, the self-learning technique cannot be applied.

The ER process is triggered by adjustment events. Events are structured data defined in the semantic model (Section 3.6) and generated by the machine or by agents, and transmitted in XML format using a publish-subscribe pattern. The ER agent subscribes to adjustments events, i.e. it listens to this type of events. Figure 2 shows an example of adjustment event serialised into XML. The various FrameKeyModel elements uniquely identify the parameter being modified by referring to entities defined in the semantic model. Experience can be created in two different ways: an experience instance can be created for each individual adjustment or for an adjustment session. In the former case, when the ER agent receives an adjustment event it queries the event base, a deductive and object oriented data store of all the generated events, for the state events generated by the machine before and after the adjustment. These events are used to construct a representation of the state of the machine before and after the adjustment. A machine state is represented by a list of attribute-value pairs in the experience base. Figure 3 shows an example of state event serialised into XML. The ER agent then calculates the KPI on both states and creates an experience instance with this information in the experience base. In the latter case (Fig. 4), a “start adjustment session” event signals that a series of adjustments is going to take place. When the ER agent receives this event, it captures the state of the machine at that point and calculates the KPI, then it records and groups all the adjustment events generated until it receives an “end adjustment session” event. At that point, the ER agent captures the state of the machine again, calculates the KPI and creates an experience instance with these two KPI values. In both cases, the representation of a machine state at a certain point in time is constructed by querying the event base for the latest state events generated up to that point. The KPI is calculated on such representation. After an adjustment occurs, it is normally necessary to wait for some parts to be manufactured before capturing the state of the machine again, calculating the KPI on the new state and thus measuring the impact of the adjustment. Figure 5 outlines the XML structure of an experience instance. MachineStateAttributeList in MachineStateBeforeChange and MachineStateAfterChange is a list of attribute-value pairs containing all the attributes of the associated machine state. FrameAdjustmentEventList in AdjustmentLog contains all the adjustments of the session if experience is created per adjustment session, and only one adjustment otherwise.

The choice between creating experience per adjustment or per adjustment session depends on the level of refinement that one wants to have in the experience base and therefore when adapting, as well as on performance considerations. If experience is created per adjustment, more specific knowledge on the impact of single adjustments will be available. For each occurrence of a certain type of adjustment, a separate evaluation with KPI calculations will take place, thus providing a series of related experience instances in different contexts. If experience is created per session, less specific knowledge on single adjustments will be available, but it will be possible to suggest multiple adjustments at each step of an adaptation process, as described in Section 4. In addition, if experience is created per adjustment, more queries to the event base will be executed (two for each adjustment), compared with the case of adjustment sessions (two for each session).

The experience base is used by the LEARN agent, described in the next subsection.

The LEARN agent implements a learning technique for generating adaptation knowledge. This technique generalises the examples contained in the experience base created by the ER agent. This knowledge is used by the ADAPT agent to recommend adjustments to the user (the engineer or system integrator making changes). The LEARN agent is invoked by the ADAPT agent during an adaptation.

The following is a step-by-step description of a typical adaptation scenario employing the LEARN and ADAPT agents, together with the other agents, in FRAME:

The ranked list of adjustments is generated as described in the Section 4. The operator’s chosen action results in a new adjustment that triggers the creation of new experience on that adjustment. Of course, the operator does not necessarily have to choose one of the presented adjustments and they are free to apply any. The new experience will refine future rankings in similar contexts.

The HMI agent provides an intelligent interface between the user and the agent framework, since it can guide the user and not simply receive input or present output. This agent plays these roles:

The framework architecture has two layers: a lower layer, called station level, and an upper layer, called system level (Fig. 6). The station level interfaces directly the machine, whereas the system level interfaces both the machine and the station level. FRAME agents are deployed on both layers. The agents of the station level are used for adapting individual modules or assembly stations and are deployed on each station. The agents of the system level are used for adapting the production system as a whole. Agents of the same type can, therefore, run simultaneously both at the station level in different stations and at the system level. Furthermore, other components (event base, experience base, semantic model and behaviour model) are also deployed on both layers.

In general, agents of the same type have the same role and behaviour in the two layers, but there are some important differences. The station level processes all the events generated by each individual station separately, whereas the system level receives all the events generated by all the stations. The ER agent of the system level captures the adjustments that have an effect on multiple stations, as well as those that are applied at the station level, and adds them to a system-level experience base. The system-level experience base represents all these changes, including an evaluation of their effect on the whole system. In addition, the system-level experience base contains the experience received from all the stations. A LEARN agent of the station level applies the learning algorithm only to the station-level experience base of the station it is deployed on. In contrast, the LEARN agent of the system level applies the learning algorithm to the system-level experience base. The ADAPT agent of the system level creates a representation of the system-level machine state as the union of all station-level machine states (where a machine state is represented by a set of attribute-value pairs). Note that the station level can operate independently of the system level and does not require its presence. Similarly, agents of the station level deployed on different stations do not interoperate.

Since learning can occur at both station and system level, there can be changes having an opposite effect at different levels, in the sense that they are positive for a station but negative for the whole system, or the other way around. In general, the rankings of station-level adjustments may not be the same at the system-level. Also, a system-level change can have different effects on different stations. Clearly, the objective of an adaptation is to reach a target KPI at the system level. Although we do not have mechanisms to relate precisely this global objective to local ones, in order to reach the global objective, it is typically useful to focus on local objectives in the first instance. This is especially the case if there is more experience available at the station level than at the system level, indicating that the changes recommended at the station level are more reliable. This happens, in particular, when an adaptation involves using a module, possibly from another system, for which an experience base has already been built, whereas there is not experience available yet for the other modules, stations or for the full system.

An important part of the framework is composed of two models: a common semantic model (CSM) and a (system) semantic model (SM). The CSM is a meta-model of a production system, that is, a general model that describes what a typical automated production system consists of (resources, capabilities, devices, operations, products, parts, etc.), but does not define a particular one. The production engineer uses the CSM to create a SM for a specific production system, in which all the application-dependent declarative knowledge is represented. One particular application of the CSM is structuring operator knowledge and relating it to machine data during the ramp-up of an assembly system [32]. Although the main use case of FRAME is assembly systems, the default CSM provided by FRAME can be extended to cater for different manufacturing processes.

In particular, a SM for an assembly system specifies the properties of products being assembled, assembly process and equipment, as well as the relationships between them. Specifically, the SM contains parts and products, assembly operations and associated modules, process parameters, sensor data, performance measures, agents and their capabilities. The combination of CSM and SM provides a knowledge representation framework for modelling sensor data, experience and adaptation knowledge, that is also used for agent communication. Agents can produce or consume data whose structure is defined in the CSM/SM or data that reference entities defined in the CSM/SM.

Both CSM and SM are created using the ObjectLogic language [4] and the OntoBroker semantic middleware [33]. ObjectLogic is a deductive, object-oriented database language which combines a declarative semantics and an object-oriented data model. OntoBroker enables FRAME to store a large number of events in an event base and execute very expressive queries written in ObjectLogic. Events specify properties or instantiate elements defined in the SM, whereas queries are used to retrieve events. For example, Fig. 7, shows a query that retrieves all the latest events related to the same SM element generated in a specified time interval.

Both events and queries can reference elements defined in the CSM/SM. An editor is provided to create a SM using a mind map approach, so that no knowledge of ObjectLogic is required. The mind map is translated into ObjectLogic automatically, and the generated SM can be edited manually if necessary. An extract of the mind map of the SM used in the experiment (Section 5) is shown in Fig. 8. A screen capture of the SM editor is shown in Fig. 13 in the Appendix.

Based on the SM, a behaviour model using discrete event simulation can be defined to analyse the behaviour of the production line. A behaviour model allows to simulate changes that are proposed by the ADAPT agent or that the operator intends to try out. Such a model is useful to detect poor potential changes without actually applying them on the physical machine. Key performance indicators that can be calculated include cycle time, machine and buffer usage, and failure rate. Simulations results produce knowledge on the effect of changes. This form of knowledge can be used when there is no information about a particular change in the experience base.

The creation of a behaviour model from a semantic model can be automated and the model can be dynamically updated to reflect the current state of the modelled system when changes have been applied, as presented by [34]. A behaviour model can be used by the ADAPT agent or by the user. Behaviour models are implemented in FRAME using the simulation modelling software Lanner Witness.

The search in the experience base is based on a variant of the k-nearest neighbour classification algorithm (kNN). Adaptation contexts are represented by points in a multidimensional space and their similarity is captured by a distance function. In kNN, given a point x to classify, the k nearest points to x are located and the class to assign to x is chosen among the neighbours’ classes by applying a voting scheme. A simple voting scheme consists of assigning the most common class among the neighbours (see Fig. 9). The class of an adaptation context is the adjustment being applied in it.

Adaptation contexts can be defined by numerical or categorical attributes. The value of an attribute can be undefined if, for example, the module or sensor that produces it is not present or is faulty. Hence, we use a heterogeneous Euclidean-overlap metric (HEOM) [35]: where: The function overlap gives 0 if its arguments are the same, otherwise 1. The function rndiff_i (range normalised difference) is defined as: where \(\max \limits _{i}\) and \(\min \limits _{i}\) are, respectively, the maximum and minimum values observed in the training set for attribute i.

In the generation of the rankings, we consider not only the similarity between adaptation contexts, but also the system performance obtained by the adjustments in the experience base. The performance is measured by the KPI of the machine state after the adjustment. An experience instance has the form \((\mathbf {x},\mathsf {adj},\mathbf {x}^{\prime })\), where: Let The similarity-performance function \({e^{d}_{f}}\) [7] is defined as: The smaller the value of \({e^{d}_{f}}(E,\mathbf {y})\), the better adj is anticipated to be in y.

Let y be an adaptation context to be classified, d a distance function and f a KPI. There are two cases to consider: