Temporal fusion transformer-based prediction in aquaponics

This study utilized the sensor-based aquaponics dataset proposed by Ogbuokiri et al. [20]. This dataset was selected because it is a recent dataset, has high-quality data points with reliable sensor measurements, includes several relevant parameters, is easily accessible, and has a suitable frequency of data collection. The prediction of nitrate levels in aquaponic systems was performed for the first time using a time-stamped dataset with a high-dimensional feature space. The dataset contains 6 parameters of water quality sensors (i.e., temperature, turbidity, dissolved oxygen, pH, ammonia, nitrate), time, and physical conditions of fish (i.e., length, width, population). The dataset’s default data collection interval is five seconds and contains sensor data for nine freshwater catfish ponds. Each sensor was initially calibrated in accordance with industry standards before being tested [20]. The trustworthiness of the data and the lack of a prior time-stamped investigation led to the selection of the proposed dataset. The basic statistical analysis of dataset parameters is depicted in Table 2.

Normalization is required if the features have drastically different values. The Aquaponics dataset shows a high variational difference between parameters, which leads to some features being dominant, as shown in Fig. 3. The proposed features are normalized to similar sizes because they should be equally important for estimating the nitrate level in an aquaponic system analysis. Without normalization, the training model could blow up with NaNs if the gradient update is too large. By defining a unique effective learning rate for each feature, optimizers like Adagrad and Adam provide protection against this problem. Still, ELM is not a gradient-based algorithm, meaning that the optimizer functionality cannot be utilized. Unlike gradient-based models, high-variance input data causes input saturation in the ELM model, where the activation function gets saturated at spiked values, limiting the model’s capacity to understand the underlying patterns in the data. This can be addressed by applying normalization techniques to the dataset. According to studies [21, 22], min-max normalization performs better than its counterparts in time-series-based analysis. Each piece of input is normalized to a value between 0 and 1, which minimizes the impact of noise and guarantees that neural networks update parameters effectively, accelerating the training of the network. Therefore, min-max normalization is utilized in the study as given in Eq. [1] to normalize the features.X shows the input variable; min and max values point to the lowest and highest points present in the series, and \(\widetilde{X}\) indicates the normalized value. The dataset also contained missing values with a percentage of around 0.001%. These missing values have been cleaned from the dataset. The original dataset reports at an interval of 20 s, which leads to a noisy structure. Thus, the interval length has been selected as 60 s for the study, effectively merging each of the three sensor reports by using the arithmetical mean operator. There are 421140 data points in chronological order. There are three subsets of the entire data set: the training dataset, the validation dataset, and the test dataset. The data set consists of 90% training data, 8% validation data, and 2% test data.

RNN and ELM differ in that RNN is better at handling sequence data. It has the ability to back-propagate as well. The gradient’s functionality is to update the recurrent neural network’s weight value. If the weight is kept too low, the gradient vanishes, and the hidden layer’s ability to learn diminishes. The gradient also explodes if the specified weight is too high. RNNs also include feedback connections in the hidden layer units of their architecture. They are able to process temporal information and learn sequences as a result of this capability. The hidden layer functions as a memory and has the capacity to store sequential data.

The LSTM method was developed as an improved version of RNNs, where the vanishing gradient problem is addressed [23]. The term “gated” cell refers to this type of cell because it allows the user to choose whether to retain or disregard stored information. LSTM comprises three gates, namely an input gate, a forget gate, and an output gate (Fig. 4).

Forget gate selectively decides what information from earlier time steps should be retained. Equation 2 is applied.In order to determine which information should be retained in the LSTM memory, this control gate employs a sigmoid function. The values of \(h(t-1)\) and x(t) are largely responsible for the selection. f(t) produces output values between 0 and 1. The values close to 0 denote the total loss of the previously acquired information, while the values close to 1 preserve the entire information.

Input gate that determines which information from the most recent time step should be added. According to Eqs. 3, 4 and 5, this gate is made up of a sigmoid layer and a hyperbolic tangent (tanh) layer.A vector of new candidate values that will be added to the LSTM memory is represented by \(i_{2}\) and \(i_{1}\) represents whether the value needs to be modified or not. Following that, element-wise multiplication is applied to the tanh and sigmoid outputs. Cell state, which carries information during the entire sequence and in Eq. 6, serves as a representation of the network’s memory.First, the forget gate’s output is multiplied element-wise by the cell state from the previous time step. This makes it possible to reject values in the cell state when they are multiplied by values close to 0. Next, the cell state is enhanced by adding the input gate’s output element by element. The new cell state is what is produced in Eq. 7. The output gate decides the value of the output at the current time step.This gate first determines which side of the LSTM memory contributes to the output using a sigmoid layer. The information that the hidden state should contain is ultimately determined by multiplying the tanh output by the sigmoid output in Eq. 8. The new hidden state is the output.

The Encoder–Decoder standard model [24] is generally incapable of accurately handling long input sequences. The encoder processes the input sequence and compresses the information into a context vector of fixed length. Therefore, only the last hidden state of the encoder RNN is used as the context vector for the decoder. It is expected that this representation will be a good summary of the full capture sequence. On the contrary, the first part is mostly forgotten once it completes the processing of the entire entry. In the Encoder–Decoder model, an encoder reads the input sentence, a sequence of vectors \(x = (x_{1}\),...,\(x_{T})\), into a vector c. At each time step t, the hidden state \(h_{t}\) of the RNN is updated by using Eqs. 9 and 10 where \(f\) and \(q\) are nonlinear activation functions.The suggested model’s decoder is conditioned to produce the output sequence by anticipating the subsequent symbol \(y_{t}\) given the hidden state \(h_{t}\). Additionally, \(y_{t}\) and \(h_{t}\) are dependent on \(y_{t-1}\) and the input sequence’s summary \(c\).Consequently, the decoder’s hidden state at time \(t\) is computed.

Initially, attention focused on solving the main problem around the Encoder–Decoder model and achieved great success. The attention was presented by Bahdanau [25], who revolutionized the field of deep learning with the concept of parallel treatment of words instead of processing them sequentially. The central idea of this layer is as follows: Each time the model predicts an output word, it only uses parts of the input where the most relevant information is concentrated, rather than the whole sequence. It only pays attention to the most relevant inputs and the computing process, as follows:

The input sentence is mapped by an encoder to a sequence of annotations \( (h_{1}\), ...,\( h_{T}) \), which determine the context vector \(c_{i}\). Although each annotation \(h_{i}\) contains details on the input sequence, only a specific portion of the input is highlighted. As a weighted sum, the context vector is subsequently calculated as follows:This is known as the alignment model. Based on how closely the input at position \(i\) and the output at position \(t\) match, the alignment model gives a score \(e_{t, i}\). The weights \(a_{t, i}\) are computed in Eq. 13 by applying a softmax operation to the previously computed alignment scores.Each time step, the decoder receives a distinct context vector, \(c_{t}\). It is calculated using the weighted total of all the hidden states of the encoder in Eq. 14.

The TFT provides a neural network architecture that combines the workings of a number of existing neural architectures, such as LSTM layers, encoder–decoders, and the attention heads used in transformers, as shown in Fig. 5 [26]. The transformer primarily consists of an encoder and a decoder, where the encoder part uses the time series data as input and the decoder part produces context-aware embeddings to predict future values. LSTM encoders and decoders summarize shorter patterns, whereas long-range relationships are left to the attention heads. The temporal multi-head attention block finds and prioritizes the most important long-range patterns that the time series may include. Each attention head can focus on a different temporal pattern.

The context vector is supplied to the Gate layer and then, to the Add & Norm layer. The dropout layer is only used during training and helps prevent the over-fitting of the network by randomly eliminating some weights at a rate set by the user. The gated layer merely regulates the bandwidth of information flow within a particular neuron. Self-attention gathers information from a couple of different neurons. The layer first combines the weights from the gated layer with the residual connection weights in the Add & Norm layer. The dependency on batches is eliminated by later normalizing each input to a particular layer across all features. Because of this feature, sequence models like transformers and recurrent neural networks are well suited for layer normalization.

In terms of processing and predicting time series data, TFT models have proven to be more sophisticated than conventional LSTM models. By taking advantage of self-attention, this model offers a novel multi-head attention mechanism that, when analyzed, sheds further light on feature significance. Therefore, in contrast to other deep neural networks, these features are no longer regarded as black box. The following is a list of TFT’s primary components:

Backpropagation employs gradients as a basis. The structural nature of the algorithm provides high computational capacity to model complex problems. Gradient-based neural network algorithms also show a high predisposition to local optimus. Thus, regarding the nature of the problem analyzed, the algorithm is also included in the study for comparative purposes.

The Extreme Learning Machine (ELM) offers a rapid and powerful alternative for both Machine Learning and Deep Learning-based solutions [27]. ELM is a training approach for a single hidden layer feed-forward neural network (SLFN). The architecture employs the following three layers: an input layer, a hidden layer, and an output layer. The hidden layer bias and input weights for the Extreme Learning Machine are determined at random and frozen during training. The ELM only optimizes the hidden layer weights. A single training iteration and random hidden layer weights enable faster convergence to the global optimum.

Mathematically, ELM can be formulated according to the following equation:

The development language for both pre-processing and network implementation was Python version 3.7. The proposed LSTM networks were implemented using the Keras framework with version 2.9.0, and the TFT network was developed with Pytorch Lightning version 1.8.0.post1 and Pytorch Forecasting version 0.10.1.

The hyperparameter optimization has been conducted at the B.T.U. High-Performance Clustering Laboratory (HPCLAB).¹ The model is trained on Nvidia 3090 GPUs with CUDA version 8. Eight GPUs were used in parallel to accelerate the overall training process.

Three measures, namely Mean Absolute Error (MAE), Explained Variance Score, and Mean Square Error (MSE), were used to evaluate the proposed model’s predictive ability. The Mean Absolute Error is the difference between the expected and actual values expressed in absolute terms. In MSE, the average squared difference between the observed and predicted values is assessed. Any unfavorable indications are changed by the squaring. The error measures are defined in Eqs. [31] and [32].Explained Variance Score is a metric for measuring the disparity between a model’s predictions and the actual data. In other words, it is the portion of the model’s total variance that is not attributable to error variance but is explained by factors that are actually present. Scores close to 1.0 are highly desired, and error measures are defined in Eqs. [33]. The variance of the predicted errors and the variance of the actual values are denoted by \( Var(y_{a}-y_{p} ) \) and \( Var(y_{a}) \), respectively.\( R^2 \) Eqs. [34], where N is the total number of the forecasting value, \(y_{p}\) is the predicted value, \(y_{a}\) is the original actual value, and \(y_{average}\) is the average of the original value. Perfect forecasting results in a value of 1, whereas a value of 0 means that the performance is identical to that of a simple model that consistently forecasts the mean value of the data. There is minimal association between the findings and the dataset when the \( R^2 \) value is negative.

The selection of hyperparameters has a significant impact on how well deep learning models perform. Because of this, fine-tuning becomes crucial in the training stage to produce a successful model. This work uses the Keras Bayesian optimizer within the Keras-Tuner framework [28] to construct a hyperparameter optimization utilizing a random search method for the LSTM models. The ELM hyperparameters are optimized using a hand-made search technique. The hyperparameters for the suggested TFT network for predicting aquaponics systems have also been determined using Optuna [29] optimizer techniques within the Pytorch framework. The resulting combinations are shown in Table 3. Random combinations of all accessible hyperparameters defining the search spaces are produced, with the number of combinations generated depending on the maximum number of trials and the number of models to be trained for each test. A model is trained for each of these combinations, and the one that performs the best is saved as the best model.

Table 4 and Fig. 7 present the error rates of the proposed study for different methods concerning the metrics RMSE, MAE, Explained Variance and \(R^{2}\). The proposed TFT algorithm demonstrates remarkable improvements over baseline models in all metrics. The higher rates of Explained Variance represent better association of variance between the original data-space and the generated data-space.

In all time windows, the performance improvement is also competitive. As the measurements deteriorate, the proposed model’s limitation of a one-hour forecasting window remains. The proposed TFT algorithm performs better than all baseline methods over an all period. As the value of the forecasting period rises, the forecasting performance of all models gradually deteriorates.

The proposed model also shows clear improvements in terms of metric scores over the previous works. Table 5 shows a comparison of the proposed study with similar studies.

In the aquaponics nitrate forecasting study, having lots of training data can improve accuracy and produce more effective outcomes. Training, validation, and test rates must be very carefully adjusted in order to prevent over-fitting. Otherwise, the likelihood of receiving inaccurate forecasts is high. Figure 7 displays the estimation graph that the entire method generated. The test results made using the hourly nitrate data gathered for the following all test data are displayed.

Short-term representational strength is lacking in the baseline models. Figure 7 shows that the models can pick up certain aspects of the long-term structure. Although the dataset’s characteristics have been normalized, the internal structure of the dataset poses a complex modeling challenge for the baseline models. Because of the sudden jumps it contains, the basic models fail to simulate temporal representation. The result is that the look-ahead output is skewed and vulnerable to temporal drift. Shifts were always present in the forecasts, despite the ELM and LSTM models in Fig. 7 being able to detect the noise. However, alternative algorithms produce forecasts with a smoother transition while not capturing noisy data. This guarantees that the models produce results that are accurate. When looking at Fig. 7e, it is clear that the TFT’s predicted and actual nitrate level values overlap and that there is no excess in the deviations or variances between them. Given the near values of the data and the similarity of the directional breaks, it is clear why the graph is successful.

The results of the various temporal test data sets of the proposed model are shown in Fig. 8, and it is clear that even though it does not exactly match the data, the model enables the noise to be estimated with a smooth transition. The model is quite good at forecasting long-term predictions, but due to the enormous noise in short-term projections, it cannot capture such predictions accurately. However, the short-term model can sufficiently describe the pattern.

The computational performance of the models can be examined through the models’ parameter sizes or the total time spent on the training phase [14]. In terms of both model parameter size and total training time (Table 6), the suggested TFT model is higher to the basic models. The Encoder–Decoder layers operate with much fewer parameters while providing the model with improved learning capacity. The proposed model needs more epochs to train properly and requires more time in total. Prediction timeframes are the longest of all methodologies yet provide for sufficient time to run in production settings. The proposed TFT model outperforms existing algorithms in real-world settings in terms of performance needs.