Top

Granular Computing

Published in:

11-06-2023 | ORIGINAL PAPER

A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization

Authors: Turan Cansu, Emine Kolemen, Özlem Karahasan, Eren Bas, Erol Egrioglu

Published in: Granular Computing | Issue 6/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Long short-term memory deep artificial neural network is the most commonly used artificial neural network in the literature to solve the forecasting problem, and it is usually trained with the Adam algorithm, which is a derivative-based method. It is known that derivative-based methods are adversely affected by local optimum points and training results can have large variance due to their random initial weights. In this study, a new training algorithm is proposed, which is less affected by the local optimum problem and has a lower variance due to the random selection of initial weights. The proposed new training algorithm is based on particle swarm optimization, which is an artificial intelligence optimization method used to solve the numerical optimization problem. Since particle swarm optimization does not need the derivative of the objective function and it searches in the random search space with more than one solution point, the probability of getting stuck in the local optimum problem is lower than the derivative algorithms. When the proposed training algorithm is based on particle swarm optimization, the probability of getting stuck in the local optimum problem is very low. In the training algorithm, the restart strategy and the early stop condition are included so that the algorithm eliminates the overfitting problem. To test the proposed training algorithm, 10-time series obtained from FTSE stock exchange data sets are used. The proposed training algorithm is compared with Adam’s algorithm and other ANNs using various statistics and statistical hypothesis tests. The application results show that the proposed training algorithm improves the results of long short-term memory, it is more successful than the Adam algorithm, and the long short-term memory trained with the proposed training algorithm gives superior forecasting performance compared to other ANN types.

previous article Intuitionistic fuzzy entropy-based knowledge and accuracy measure with its applications in extended VIKOR approach for solving multi-criteria decision-making

next article A decision-making mechanism for multi-attribute group decision-making using 2-tuple linguistic T-spherical fuzzy maximizing deviation method

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Arunkumar KE, Kalaga DV, Kumar CMS, Kavaji M, Brenza TM (2022) Comparative analysis of Gated Recurrent Units (GRU), long Short-Term memory (LSTM) cells, autoregressive Integrated moving average (ARIMA), seasonal autoregressive Integrated moving average (SARIMA) for forecasting COVID-19 trends. Alex Eng J 61(10):7585–7603

Barrera Animas AY, Oyedele LO, Bilal M, Akinosho TD, Delgado JMD, Akanbi LA (2022) Rainfall prediction: a comparative analysis of modern machine learning algorithms for time-series forecasting. Mach Learn Appl 7:100204

Bas E, Yolcu U, Egrioglu E (2021) Intuitionistic fuzzy time series functions approach for time series forecasting. Granul Comput 6:619–629

Bas E, Egrioglu E, Karahasan O (2022a) A Pi-Sigma artificial neural network based on sine cosine optimization algorithm. Granul Comput 7:813–820

Bas E, Egrioglu E, Kolemen E (2022b) Training simple recurrent deep artificial neural network for forecasting using particle swarm optimization. Granul Comput 7(2):411–420

Bi JW, Liu Y, Li H (2020) Daily tourism volume forecasting for tourist attractions. Ann Tour Res 83:102923

Bilgili M, Arslan N, Şekertekin A, Yaşar A (2022) Application of long short-term memory (LSTM) neural network based on deep learning for electricity energy consumption forecasting. Turk J Electr Eng Comput Sci 30(1):140–157

Cen Z, Wang H (2019) Crude oil price prediction model with long short-term memory deep learning based on prior knowledge data transfer. Energy 169:160–171

Chen SM, Jian WS (2017) Fuzzy forecasting based on two-factors second-order fuzzy-trend logical relationship groups, similarity measures and PSO techniques. Inf Sci 391:65–79MathSciNet

Chen J, Xing H, Yang H, Xu L (2018) Network traffic prediction based on LSTM networks with genetic algorithm. In: Proceedings of International Conference on Signal and Information Processing, Networking and Computers, pp 411–419

Chen SM, Zou XY, Gunawan GC (2019) Fuzzy time series forecasting based on proportions of intervals and particle swarm optimization techniques. Inf Sci 500:127–139MathSciNet

Chimmula, Zhang (2020) Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos Solit. Fractals 135:109864MATH

Chung H, Shin KS (2018) Genetic algorithm-optimized long short-term memory network for stock market prediction. Sustainability 10(10):3765

Dong D, Li XY, Sun FQ (2017) Life prediction of jet engines based on LSTM-recurrent neural networks. Prognostics And System Health Management Conference, pp 1–6

Du B, Huang S, Guo J, Tang H, Wang L, Zhou S (2022a) Interval forecasting for urban water demand using PSO optimized KDE distribution and LSTM neural networks. Appl Soft Comput 122:108875

Du J, Zheng J, Liang Y, Lu X, Klemes JJ, Varbanov PS, Shahzad K, Rashid MI, Ali AM, Liao Q, Wang B (2022b) A hybrid deep learning framework for predicting daily natural gas consumption. Energy 257:124689

Egrioglu E, Fildes R, Bas E (2022) Recurrent fuzzy time series functions approaches for forecasting. Granul Comput 7:163–170

Ensafi Y, Amin SH, Zhang G, Shah B (2022) Time-series forecasting of seasonal items sales using machine learning–A comparative analysis. Int J Inf Manag Data Insights 2(1):100058

Fan MH, Chen MY, Liao EC (2021) A deep learning approach for financial market prediction: utilization of Google trends and keywords. Granul Comput 6:207–216

Firouzjaee and Khaliliyan (2022) LSTM architecture for oil stocks prices prediction. arXiv

Fischer T, Krauss C (2018) Deep learning with long short-term memory networks for financial market predictions. Eur J Oper Res 270(2):654–669MathSciNetMATH

Freeboroug, Zyl (2022) Investigating explainability methods in recurrent neural network architectures for financial time series data. Appl Sci 12(3):1427

Gong Y, Zhang X, Gao D, Li H, Yan L, Peng J, Huang Z (2022) State-of-health estimation of lithium-ion batteries based on improved long short-term memory algorithm. J Energy Storage 53:105046

Gundu V, Simon SP (2021) PSO–LSTM for short-term forecast of heterogeneous time series electricity price signals. J Ambient Intell Humaniz Comput 12:2375–2385

Hocreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

Huang R, Wei C, Wang B, Yang J, Xu X, Wu S, Huang S (2022) Well performance prediction based on long short-term memory (LSTM) neural network. J Pet Sci Eng 208:109686

Jiang L, Hu G (2018) Day-ahead price forecasting for electricity market using long-short term memory recurrent neural network. In: Proceedings of IEEE International Conference on Control, Automation, Robotics and Vision, pp 949–954

Karasu S, Altan A (2022) Crude oil time series prediction model based on LSTM network with chaotic Henry gas solubility optimization. Energy 242:122964

Karevan, Suykens (2020) Transductive LSTM for time-series prediction: an application to weather forecasting. Neural Netw 125:1–9

Kilinc HC, Yurtsever A (2022) Short-term streamflow forecasting using hybrid deep learning model based on grey wolf algorithm for hydrological time series. Sustainability 14(6):3352

Kim TY, Cho SB (2019) Predicting residential energy consumption using CNN-LSTM neural networks. Energy 182:72–81

Kingma D, Ba JL (2014) Adam: a method for stochastic optimization. Comput Sci 1–15. arXiv: 1412.6980

Kumar G, Singh UP, Jain S (2022a) An adaptive particle swarm optimization-based hybrid long short-term memory model for stock price time series forecasting. Soft Comput 26(22):12115–12135

Kumar S, Chitradevi D, Rajan S (2022b) Stock price prediction using deep learning LSTM (long short-term memory). In: Proceedings of International Conference on Advance Computing and Innovative Technologies in Engineering, pp 1787–1791

Li Q, Yang Y, Yang L, Wang Y (2022) Comparative analysis of water quality prediction performance based on LSTM in the Haihe River Basin. China Environ Sci Pollut Res 30(3):7498–7509

Liu Y, Song M (2022) Few samples learning based on granular neural networks. Granul Comput 7:572–589

Liu H, Mi XW, Li YF (2018) Wind speed forecasting method based on deep learning strategy using empirical wavelet transform, long short-term memory neural network and Elman neural network. Energy Convers Manag 156:498–514

Liu K, Zhou J, Dong (2021) Improving stock price prediction using the long short-term memory model combined with online social networks. J Behav Exp Finance 30:100507

Livieris LE, Pintelas E, Pintelas P (2020) A CNN–LSTM model for gold price time-series forecasting. Neural Comput Appl 32(23):17351–17360

Moalla H, Elloumi W, Alimi AM (2017) H-PSO-LSTM: Hybrid LSTM trained by PSO for online handwriting identification. In: Proceedings of International Conference on Neural Information Processing, pp 41–50

Ning Y, Kazemi H, Tahmasebi P (2022) A comparative machine learning study for time series oil production forecasting: ARIMA, LSTM, and Prophet. Comput Geosci 164:105126

Pant M, Kumar S (2022a) Fuzzy time series forecasting based on hesitant fuzzy sets, particle swarm optimization and support vector machine-based hybrid method. Granul Comput 7:861–879

Pant M, Kumar S (2022b) Particle swarm optimization and intuitionistic fuzzy set-based novel method for fuzzy time series forecasting. Granul Comput 7(2):285–303

Peng L, Zhu Q, Lv SX, Wang L (2020) Effective long short-term memory with fruit fly optimization algorithm for time series forecasting. Soft Comput 24:15059–15079

Pirani M, Thakkar P, Jivrani P, Bohara MH, Garg D (2022) A comparative analysis of arima, gru, lstm and bilstm on financial time series forecasting. In Proceedings of IEEE International Conference on Distributed Computing and Electrical Circuits and Electronics, pp 1–6

Qiu YY, Zhang Q. Lei M (2020) Forecasting the railway freight volume in China based on combined PSO-LSTM model. In: Proceedings of Journal of Physics: Conference Series, pp 012029

Quadir MdA, Kapoor S, Chris Junni AV, Sivaraman AK, Tee KF, Sabireen H, Janakiraman N (2022) Novel optimization approach for stock price forecasting using multi-layered sequential LSTM. Appl Soft Comput 134:109830

Senanayake S, Pradhan B, Alamri A, Park HJ (2022) A new application of deep neural network (LSTM) and RUSLE models in soil erosion prediction. Sci Total Environ 845:157220

Seviye L, Kong W, Qi J, Zhang J (2018) An improved long short-term memory neural network for stock forecast. In MATEC web of conferences, pp 01024

Shao B, Li M, Zhao Y, Bian G (2019) Nickel price forecast based on the LSTM neural network optimized by the improved PSO algorithm. Math Probl Eng 2019:1934796

Shin Y, Gosh J (1991) The Pi-Sigma network: an efficient higher order neural network for pattern classification and function approximation. In: Proceedings of the International Joint Conference on Neural Networks, Seattle, pp 13–18

Silka J, Wieczorek M, Wozniak M (2022) Recurrent neural network model for high-speed train vibration prediction from time series. Neural Comput Appl 34:1–14

Singha D, Panse C (2022) Application of different machine learning models for supply chain demand forecasting: comparative analysis. In: Proceedings of International Conference on Innovative Practices in Technology and Management, pp 312–318

Sirisha UM, Belavagi MC, Attigeri G (2022) Profit prediction using ARIMA, SARIMA and LSTM Models in time series forecasting, a comparison. IEEE Access 10:124715–124727

Solgi R, Loaciga HA, Kram M (2021) Long short-term memory neural network (LSTM-NN) for aquifer level time series forecasting using in-situ piezometric observations. J Hydrol 601:126800

Stajkowski S, Kumar D, Samui P, Bonakdari H, Gharabaghi B (2020) Genetic-algorithm-optimized sequential model for water temperature prediction. Sustainability 12(13):5374

Tang Y, Song Z, Zhu Y, Yuan H, Hou M, Ji J, Tang C, Li J (2022) A survey on machine learning models for financial time series forecasting. Neurocomputing 512:363–380

Tian C, Ma J, Zhang C, Zhan P (2018) A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network. Energies 11(12):3493

Veeramsetty V, Chandra DR, Salkuti SR (2021) Short-term electric power load forecasting using factor analysis and long short-term memory for smart cities. Int J Circuit Theory Appl 49(6):1678–1703

Vidya GS, Hari VS (2020) Gold price prediction and modelling using deep learning techniques. In: Proceedings of IEEE Recent Advances in Intelligent Computational Systems, pp 28–31

Weng C, Liu S, Yang X, Peng L, Li X, Hu Y, Chi T (2019) A novel spatiotemporal convolutional long short-term neural network for air pollution prediction. Sci Total Environ 654:1091–1099

Yadav A, Ja CK, Sharan A (2020) Optimizing LSTM for time series prediction in Indian stock market. Procedia Comput Sci 167:2091–2100

Yao S, Lua L, Peng H (2018a) High-frequency stock trend forecast using LSTM Model. In: Proceedings of IEEE International Conference on Computer Science Education, pp 1–4

Yao Y, Han L, Wang J (2018b) LSTM-PSO: Long short-term memory ship motion prediction based on particle swarm optimization. In: Proceedings of the IEEE CSAA Guidance, Navigation and Control Conference (CGNCC), pp 1–5

Yazici I, Beyca OF, Delen D (2022) Deep-learning-based short-term electricity load forecasting: a real case application. Eng Appl Artif Intell 109:104645

Yu L, Liang S, Chen R, Lai KK (2022) Predicting monthly biofuel production using a hybrid ensemble forecasting methodology. Int J Forecast 38(1):3–20

Zeng S, Chen SM, Teng MO (2019) Fuzzy forecasting based on linear combinations of independent variables, subtractive clustering algorithm and artificial bee colony algorithm. Inf Sci 484:350–366

Zha W, Liu Y, Wan Y, Luo R, Li D, Yang S, Xu Y (2022) Forecasting monthly gas field production based on the CNN-LSTM model. Energy 260:124889

Zhang Y, Yang S (2019) Prediction on the highest price of the stock based on PSO-LSTM Neural Network. In: Proceedings of International Conference on Electronic Information Technology and Computer Engineering, pp 1565–1569

Zhang J, Zhu Y, Zhang X, Ye M, Yang J (2018) Developing a long short-term memory (LSTM) based model for predicting water table depth in agricultural areas. J Hydrol 561:918–929

Zhang H, Xu J, Qian L, Qiu J (2021) Prediction of the COVID-19 spread in China based on long short-term memory network. J Phys 2138:012015

Title: A new training algorithm for long short-term memory artificial neural network based on particle swarm optimization
Authors: Turan Cansu
Emine Kolemen
Özlem Karahasan
Eren Bas
Erol Egrioglu
Publication date: 11-06-2023
Publisher: Springer International Publishing
Published in: Granular Computing / Issue 6/2023
Print ISSN: 2364-4966
Electronic ISSN: 2364-4974
DOI: https://doi.org/10.1007/s41066-023-00389-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 6/2023

Solution of the Pythagorean fuzzy wave equation with Pythagorean fuzzy Fourier sine transform

Novel fuzzy similarity measures and their applications in pattern recognition and clustering analysis

Pythagorean fuzzy cognitive analysis for medical care and treatment decisions

Bootstrapped Dendritic Neuron Model Artificial Neural Network for Forecasting

Improved rough approximations based on variable J-containment neighborhoods

A MAGDM model based on 2-tuple linguistic variables and power Hamacher aggregation operators for optimal selection of digital marketing strategies

Premium Partner