Predicting performance indicators with ANNs for AI-based online scheduling in dynamically interconnected assembly systems

The market demand for highly customized products is rising and product life-cycles kept shortening over the last decades [1]. In manufacturing this trend is termed mass customization. Among others, one upcoming additional requirement is more flexibility in terms of shorter manufacturing system response times due to frequent product changes [2]. The resulting increase in complexity and system dynamics poses a challenge for production systems; flexible and fast adaptation to constantly changing market requirements is a key competitive factor [3]. Adaption of production scenarios has a high efficiency impact especially on the assembly systems, due to the increase and accumulation of complexity along the value chain and the number of sub-processes that are required to be conducted [4]. Hence, high variety product portfolios can become economically feasible in the final assembly by using adaptable systems configurations [5, 6].

In addition to the market side, on the manufacturing side, the organization of assembly systems advances as a reaction to the described additional market requirements. Especially the layout and the material flow are affected by these new organizational forms , described in the following, as they are derived from the concept of job-shop manufacturing [7]. Restrictions of a linear transfer and a super-ordinate cycle time are omitted to create higher degrees of freedom in the job route, which is the sequence of operations performed on the product at a sequence of resources [8]. This concept is coined under the terms of matrix assembly [9], modular assembly systems [10] or dynamically interconnected assembly systems (DIAS) [11]. DIAS is taken as a reference system in the following , since it suits most to the dynamical challenges that today’s production systems are facing, as illustrated above.. Since the job route has increased degrees of freedom, the production planning control is especially affected due to higher solution spaces and computational complexity by the upcoming organizational forms of assembly systems.

From the computer science domain, digital technologies are pushing beyond the existing level of manufacturing execution [12] and enable IT systems to plan and control mass customization and the production of highly customized products [13]. Production planning and control generally comprise the determination of product quantities to be produced in accordance with an optimization of the enterprise’s objective such as productivity, profitability, and adherence to due dates [14]. Also, it includes steering of the production processes, the synchronization of stations, and the allocation of stations to tasks over given time periods [15, 16]. The latter is attributed to production scheduling, which is especially influenced by production turbulences, which results in the need of time-sensitive decision-making to quickly react to the turbulences and disruptions? A resulting applied method is re-scheduling with focus on time-sensitive decision-making [17]. Concerning scheduling of complex production systems such as job shops, several review surveys show that machine learning techniques such as supervised learning are increasingly utilized to cope with the challenge of optimizing production schedules [18‐21].

Another influence is the more complex job route in DIAS, which increases the solution space for an optimal schedule. The dynamical characteristics of DIAS require timely scheduling and rapid decision-making. The speed at which the control system in DIAS needs to react is bound to real-time conditions, limiting the time-span for a response to a certain threshold [22]. Consequently, online scheduling algorithms, which are applied during the manufacturing execution phase, are suitable for production control in DIAS [23].

An outstanding example of innovations in computer science is the application of machine learning for an artificial intelligence (AI) player in strategic board games, which showed groundbreaking performance results. In 2016 Silver et al. published an AI algorithm, called AlphaGo, that achieved super-human performance in the strategy board game Go and defeated the world champion [24].

The basic components of AlphaGo are a Monte Carlo tree search (MCTS) algorithm to model the decision-making within the game and to find optimal moves. A deep artificial neural network (ANN) is trained to select moves and another ANN is trained for a rapid evaluation of board states.

In 2017 Silver et al. generalized the AlphaGo Zero approach [25], which was a follow-up version of AlphaGo, into the single algorithm AlphaZero and achieved a superhuman level in the games of go, chess, and shogi (Japanese chess). AlphaZero starts from random play without any specific expert domain knowledge except the game rules [26]. The universal algorithm architecture and the more general applicability of AlphaZero shows a high potential for use in other domains as also stated by Silver et al. [26].

DIAS require timely decision-making by computing and evaluating large solution spaces for the tasks of online scheduling. The requirements in strategic board game AI development are comparable. For instance, in a board game the reaction times are limited to a range of sub-minutes, which also applies for a broad variety of assembly scenarios, where the processes or takt times are bound to sub-minute ranges. Therefore, an application of the algorithmic architecture from AlphaZero in online scheduling is encouraging.

A critical part of the decision-making in online scheduling is the evaluation of possible decision outcomes [27]. A common approach is using discrete-event simulation (DES) to predict the outcome of a decision as multiple publications show [28‐31]. For complex systems a high computational time for the setup and execution of the simulation models is required. This is not following the needs of time-sensitive decision-making [32]. The knowledge approximation from board game AI with ANNs has its strength on time-sensitive decision-making, which is comparable to the needs of online scheduling in DIAS, and showed an extraordinary performance and is, therefore, a promising approach. The ANN offline learning of expert domain knowledge supported by simulation is the main driving factor for the success of AlphaGo and AlphaZero. For online scheduling problems, this would be equivalent to offline learning with DES. As a result, the AlphaZero ANN architecture can easily be transferred and adopted, while the expert domain knowledge in the DIAS online scheduling case is provided by established simulation models. The advances of AlphaGo Zero and AlphaZero consist of the full replacement of expert domain knowledge with simulations predicting the performance impact of game action decisions [26]. As a deduction, for online scheduling a time-sensitive method for evaluating the performance impact of possible scheduling decisions needs to be developed. Recent publications in the field of online scheduling and performance indicator prediction focus on problem types that are not directly comparable to DIAS (e.g. flow shop, sheet-metal production [21, 33, 34]) or do not take the AlphaZero architecture for the ANN input and output layer design into account (e.g. [35‐37]. Therefore, the main goal of this work is to decrease the time-span of predicting the makespan of a given system state by training and implementing an ANN that follows the architecture of the promising AlphaZero approach. This ANN is a component of a potential AlphaZero online scheduling and control algorithm that utilizes its time-sensitive prediction capabilities.

This paper is structured as follows: In Sect. 2, the boundary conditions of the production system and the online scheduling problem as well as the foundations of the AI algorithm AlphaZero will be outlined. In Sect. 3 various relevant publications are summarized and evaluated based on the requirements. Section 4 describes the implementation of a DIAS in a discrete-event simulation. This section also includes the AlphaZero-based scheduling agent, a greedy scheduling agent for comparison, and how they interact with the simulation. In Sect. 5 the data set generation and ANN training procedure are explained. In Sect. 6 the outcome of the supervised learning and hyperparameter optimization are show n and discussed. Finally, in Sect. 7 the work is briefly summarized and reflected, and further research opportunities are presented.

The foundations create the knowledge base to understand the subsequent chapters, e.g. the state of the art, the presented developments and the results. This chapter comprises explanations on the assembly organisation form of dynamically interconnected assembly systems (DIAS), the flexible job shop problem as a widespread scheduling problem type, online scheduling that is specifically fitting to DIAS and, finally, the relevant basics of AlpaZero as an board game AI.

Various publications leverage the capabilities of ANNs in different completely comparable production planning and control scenarios.

Cadavid et al. (2020) present a review on machine learning applied in production planning and control [21] on various use-cases. The most addressed use-cases are in smart planning and scheduling. Also, Cadavid et al. show that the use of ANNs for machine learning applications increased significantly in the last years. An overview of ANN applications and other supervised learning techniques is given in the publication.

A concrete approach to combine supervised learning and DES is the multi-model based prediction of throughput time by Pfeiffer et al. [33]. The described use-case is a flow shop. A DES maps a control system to generate data for the supervised learning technique. A multivariate regression that is closely related to ANNs is trained, validated, and tested on the simulation data. By inserting the statistical prediction models into a simulation-based decision support system, a control system anticipating deviations was created. The simulation was used for reactive control of disturbances as well as for training the prediction models.

Murphy et al. [35] address the problem of order due date assignment in production scheduling. For this purpose, machine learning technologies estimate job makespan in a dynamical job shop with nine stations. A DES framework was utilized to model the dynamical job shop and to provide 10,000 training data records for the machine learning algorithms. As a result, the performance of the proposed machine learning algorithms is better than the conventional due date assignment methods. Within the different machine learning methods that were added for comparison, a random forest approach performs better at high complexity and an ANN produces a higher accuracy on low complexity.

Jong et al. [36] train a multilayer perceptron ANN in a job shop and flow shop environment to quickly predict makespan for a wide range of system configurations. A simulation model serves as the data source. The input parameters are layout (job shop or flow shop), the statistical distribution of job types, processing time on a machine, and number and speed of automated transport vehicles serves as the data source. With a full factorial experiment setup and 100-fold repetition of all experiments to minimize stochastic influences, 230,400 training data sets are generated. In addition to an ANN, the three further machine learning algorithms mixture of experts, extreme learning machine, and support vector machine are trained separately for the two layout options as a reference. The ANN has the least error, but the training time is longer than for the other algorithms. The higher training time is put into perspective by the fact that an ANN is trained once with a large data set and can then be trained and improved incrementally. Therefore, multilayer perceptron ANN algorithms prove to be superior estimators compared to the other three machine learning algorithms.

Motivated by the significant progress in the field of deep learning and deep Q-learning neural networks, Waschneck et al. (2018) applied these methods to flexible job shop scheduling in the context of semiconductor production. They implement and train one deep Q-network (DQN) agent per station, which is composed of multiple machines. Each agent has the task of selecting a job from the station’s queue and allocating it to a machine. The ANN is trained to approximate a policy that selects actions given the state of the system such that the Q value as a sum of rewards is maximized. Waschneck et al. designed a reward function that incentivizes the minimization of the deviation of completion times from due dates. While the cycle times increase slightly, the DQN approach achieves a reduction of delayed jobs by up to 90% [37].

Rinciog et al. implemented a scheduling agent based on AlphaGo Zero [25] to reduce tardiness and material waste in sheet-metal production equivalent to a job shop problem. The scheduling agent allocates the different operations of each part to the machines in the production system. The agent interacts with a discrete event simulation as a surrogate for the real-world production system and receives a scheduling request whenever a machine becomes idle. Rinciog et al. adjusted AlphaGo Zero in terms of state and action space, ANN architecture, and the loss function to fit the sheet-metal production scheduling problem. The input to the ANN is a representation of the production state at the moment of the scheduling decision. The ANN consists of two fully-connected layers with 1024 and 512 neurons, respectively. The ANN has a policy head that allocates prior probabilities to all jobs, and a value head estimating the quality of the current state. Rinciog et al. defined a custom state value function that rewards reducing waste in the cutting process, punishes tardiness, and prioritizes valuable jobs. Rinciog et al. train the network in two stages. First, the network undergoes a pre-training routine, learning from the earliest due date dispatching rule. In the subsequent self-play phase the network learns from the data generated by the agent itself. To validate the AlphaGo Zero agent, the authors compared it to the earliest due date heuristic. After pre-training AlphaGo Zero and earliest due date go head to head. After reinforcement learning, AlphaGo Zero achieved the best score, with lower tardiness and less waste, in 64% of all trials, outperforming earliest due date [34].

The publications presented in this section showed how machine learning models, especially ANNs, sustain decision-making processes. They improve the scheduling agents’ performances by providing an a priori estimation of the quality of a production state or a possible action. Most of the research is focused on applications of machine learning agents to job shop and flow shop problems. An ANN especially for dynamically interconnected assembly systems was developed, which differs from most publications , where their problem type definition differs from DIAS. More specifially, Rinciog et al. [34] focus on sheet-metal production, which is not comparable to assembly systems and Rinciog et al. utilize the ANN architecture of AlphaGo Zero, whereas in this work the architecture of the newer AlphaZero is used. AlphaZero is more promising for a wider range of applications outside of the board game domain. Therefore, the approach in this work embeds the predictive qualities of ANNs, according to the AlphaZero architecture.

This section provides further details on the implementation of DIAS in a discrete-event simulation, the AlphaZero-based scheduling agent, and how they interact.

The AlphaZero agent and its ANN require a large data set for training and validation. The data set must be based on a wide range of different DIAS scenarios to prevent overfitting. The left-hand column of Table 2 lists the parameters that can be varied to generate different DIAS scenarios. A full-factorial DOE approach to vary these factors and considered the levels given in Table 2 was used.

Each level combination of the listed factors corresponds to one scenario. Multiplying the number of levels per factor, listed in the right-hand column of the table, yields the theoretical number of scenarios. In this case, it is \(34,992\). However, unrealistic combinations that would cause errors in the simulation, e.g. a scenario where stations offer fewer operations than required for the jobs , were filtered. The number of remaining scenarios is \(10,944\). The scenario analysis generates a simulation data file for each of these scenarios. Throughout the training procedure, it showed that the average number of scheduling decisions per scenario is about \(180\). Each scheduling decision is a data point consisting of a feature vector, legal actions-vector, and labels for the value and policy head. The total number of data points is roughly \(1.97*10^6\).

The number of stations and the number of operations are constant (see Table 2). These two parameters directly affect the architecture of the ANN. Altering them would require a complete rerun of the training procedure from the start. However, the agent can train its scheduling capabilities for new DIAS setups by interacting with the simulation leaving the real-world system unaffected. Once the agent is fully trained it can be deployed in the control system.

Analogous to Rinciog et al. [34] and Waschneck et al. [37] the training phase is split into two phases, pretraining (supervised learning) and self-play (reinforcement learning). Within the scope of this paper, the focus is to find a suitable ANN structure for the task at hand using the pretraining data set. The ratio of scenarios used for pretraining to those used for self-play is 2 to 1. The greedy agent presented in 4.3 converts the pretraining scenarios in training data for the supervised learning phase of the AlphaZero agent’s ANN. Each greedy scheduling decision consists of the labels for the value head and the policy head, \(z\) and \(\varvec{\pi }\), as well as the state representation \(s\) comprising the feature vector and the legal actions-vector. This corresponds to one data point. The algorithm is quite fast and requires only about 30 min to simulate the scenarios in the pretraining data set. That is a share of 40% of all generated scenarios and amounts to about \(0.78 * 10^6\) data points. The greedy algorithm is designed to minimize the period of time before a job starts its next operation. This approach also reduces the overall makespan and is a solid basis for pretraining the AlphaZero ANN.

First, a grid-search based optimization of the hyperparameters number of neurons, number of blocks, bias, and dropout rate was conducted. Then, the most promising ANN version to completion were trained. The aim is to find the ANN with the highest capability to accurately evaluate the state of the simulated DIAS and select favorable actions.

In this work, a performance prediction method with ANNs for online scheduling problems within dynamically interconnected assembly systems (DIAS) was proposed. As the dynamical environment of DIAS requires fast online decision-making, discrete-event simulation is not a possible method to be applied, due to the high needed computational times. The recent breakthroughs in AI algorithms for strategic board games present a promising solution for fast decision-making in a setup comparable to online scheduling problems. The online scheduling support task that was chosen for this work was the prediction of a performance indicator (makespan) to quickly evaluate a given system state. For this task, the advances from the AI game algorithm AlphaZero were leveraged. Specifically, an ANN that can suggest favorable job routing decisions and predicts the value of actions was created. The estimations rest upon a custom-designed feature vector and a legal actions matrix. Optimization of the ANN hyperparameters showed that a network with four layers, no dropout, no bias term, and 512 neurons per dense layer performs best. The fully trained network can predict favorable actions with an accuracy of over 95% and estimate the makespan with an error smaller than 3%.

There are several options to further develop this project and enhance the AlphaZero agent’s performance. The data set used for training can be extended. Broader value ranges of the scenario parameters increase the generalizability of the learned strategies. Switching to fractional factorial DOE scenario creation or random scenario selection can help to reduce the computational burden of training data generation while maintaining a wide diversity of scenarios. Even though it requires training a new ANN each time, the number of stations and operations should be varied beyond the values five and eight as used in this work. The increased complexity of larger DIAS layouts might benefit holistic scheduling approaches like AlphaZero.

The capabilities of the ANN could be further enhanced by including the history of the preceding states in the input. The most recent DIAS feature vectors could be stacked and serve as input to a convolutional neural network. Neural network configurations can be compared using enhanced hyperparameter tuning methods, e.g. Bayesian optimization

Throughout this work, the focus was the design of the ANN and the supervised learning stage. The parameters of the underlying MCTS were set by conducting a small-scale DOE. Further research should investigate a broader range of MCTS parameter constellations to optimize the performance of AlphaZero in flexible assembly systems. Also, future research on the subject should consider the reinforcement learning phase. The ANN requires training on data generated through self-play to surpass the heuristic it learned from in the supervised learning phase. Another open task is thoroughly benchmarking the AlphaZero scheduling agent. The algorithm must be compared to the greedy heuristic and schedules generated by an integer-linear program solving the DIAS scheduling problem optimally.