Top

Electrical Engineering

Hint

Swipe to navigate through the articles of this issue

Published in:

06-07-2022 | Original Paper

Reinforcement learning-based demand-side management by smart charging of electric vehicles

Authors: Melik Bugra Ozcelik, Mert Kesici, Necati Aksoy, Istemihan Genc

Published in: Electrical Engineering | Issue 6/2022

Abstract

In the future, the load demand due to charging of large numbers of electric vehicles (EVs) will be at such a high level that existing networks in some regions may not afford. Therefore, radical changes modernizing the grid will be required to overcome the technical and economic problems besides bureaucratic issues. Amendments to be made in the regulations on electrical energy and new tariff regulations can be considered within this scope. Smart charging of EVs is not often dealt with a solution using reinforcement learning (RL), which is one of the most effective methods for solving such decision-making problems. Most of the studies on this topic endeavor to estimate the state and action spaces and to tune the penalty coefficients within the RL models developed. In this paper, we solve the EV charging problem using expected SARSA with a novel rewarding strategy, as we propose a new approach to determine the state and action spaces. The efficacy of the proposed method is demonstrated on the problem of charging a single EV, as we compare it with a number of alternatives involving Q-Learning and constant charging approaches.

previous article A security system for frequency control based on dynamic regulation for electrical grids with renewable energy sources integrated

next article Comparison parametric and non-parametric methods in probabilistic load flow studies for power distribution networks

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Van Vliet O, Brouwer AS, Kuramochi T, Van Den Broek M, Faaij A (2011) Energy use, cost and CO2 emissions of electric cars. J Power Sources 196(4):2298–2310. https://doi.org/10.1016/j.jpowsour.2010.09.119 CrossRef

Karamuk M (2011) A survey on electric vehicle powertrain systems. In: International aegean conference on electrical machines and power electronics, ACEMP 2011 and electromotion 2011 joint conference (September), pp 315–324. https://doi.org/10.1109/ACEMP.2011.6490617

Ramachandran B, Srivastava SK, Cartes DA (2013) Intelligent power management in micro grids with EV penetration. Expert Syst Appl 40(16):6631–6640. https://doi.org/10.1016/j.eswa.2013.06.007 CrossRef

Fan H, Lu J, Li Z, Shahidehpour M, Zhang S (2019) Optimal planning of integrated electricity-gas system with demand side management. IEEE Access 7:176790–176798. https://doi.org/10.1109/ACCESS.2019.2956979 CrossRef

Russell S, Norvig P (2009) Artificial intelligence: a modern approach, 3rd edn. Prentice Hall Press, Hoboken MATH

Zhong W, Lu C, Yu R (2015) Adaptive price control for electric vehicle charging in smart grid. In: 5th International conference on information science and technology, ICIST 2015, pp 292–296. https://doi.org/10.1109/ICIST.2015.7288985

Shah J, Nielsen M, Reid A, Shane C, Mathews K, Doerge D, Piel R, Anderson R, Boulanger A, Wu L, Bhandari V, Gagneja A, Kressner A, Li X, Sarkar S (2014) Cost-optimal, robust charging of electrically-fueled commercial vehicle fleets via machine learning. In: Proceedings of 8th annual IEEE international systems conference, SysCon 2014, pp 65–71. https://doi.org/10.1109/SysCon.2014.6819237

Jinil N, Reka S (2019) Deep learning method to predict electric vehicle power requirements and optimizing power distribution. In: 5th International conference on electrical energy systems, ICEES 2019, pp 1–5. https://doi.org/10.1109/ICEES.2019.8719243

Lopez KL, Gagne C, Gardner MA (2019) Demand-side management using deep learning for smart charging of electric vehicles. IEEE Trans Smart Grid 10(3):2683–2691. https://doi.org/10.1109/TSG.2018.2808247 CrossRef

10.

Ai S, Chakravorty A, Rong C (2018) Household EV charging demand prediction using machine and ensemble learning. In: Proceedings of 2nd IEEE international conference on energy internet, ICEI 2018, pp 163–168. https://doi.org/10.1109/ICEI.2018.00037

11.

Li Z, Khaligh A, Sabbaghi N (2011) Minimum charging-cost tracking based optimization algorithm with dynamic programming technique for plug-in hybrid electric vehicles. In: IEEE vehicle power and propulsion conference, VPPC 2011, pp 1–6. https://doi.org/10.1109/VPPC.2011.6042978

12.

Lopez KL (2019) A machine learning approach for the smart charging of electric vehicles. Dissertation, Laval University

13.

Nour M, Said SM, Ali A, Farkas C (2019) Smart charging of electric vehicles according to electricity price. In: Proceedings of international conference on innovative trends in computer engineering, ITCE 2019 (February 2019), pp 432–437. https://doi.org/10.1109/ITCE.2019.8646425

14.

Wan Z, He HLH, Prokhorov D (2018) Model-free real-time EV charging scheduling based on deep reinforcement learning. IEEE Trans Smart Grid 10(5):5246–5257. https://doi.org/10.1109/TSG.2018.2879572 CrossRef

15.

Shaarbaf MR, Ghayeni M (2018) Identification of the best charging time of electric vehicles in fast charging stations connected to smart grid based on Q-Learning. In: 23rd Electrical power distribution conference, EPDC 2018, vol 3, pp 78–83. https://doi.org/10.1109/EPDC.2018.8536301

16.

Sadeghianpourhamami N, Deleu J, Develder C (2020) Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning. IEEE Trans Smart Grid 11(1):203–214. https://doi.org/10.1109/TSG.2019.2920320 CrossRef

17.

Chang F, Chen T, Su W, Alsafasfeh Q (2019) Charging control of an electric vehicle battery based on reinforcement learning. In: 10th International renewable energy congress, IREC 2019 (IREC), pp 1–63. https://doi.org/10.1109/IREC.2019.8754518

18.

Chiş A, Lundén J, Koivunen V (2017) Reinforcement learning-based plug-in electric vehicle charging with forecasted price. IEEE Trans Veh Technol 66(5):3674–3684. https://doi.org/10.1109/TVT.2016.2603536 CrossRef

19.

Vandael S, Claessens B, Ernst D, Holvoet T, Deconinck G (2015) Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market. IEEE Trans Smart Grid 6(4):1795–1805. https://doi.org/10.1109/TSG.2015.2393059

20.

Valogianni K, Ketter W, Collins J (2013) Smart charging of electric vehicles using reinforcement learning. In: AAAI workshop, Technical report WS-13-15, pp 41–48

21.

Li H, Wan Z, He H (2020) Constrained EV charging scheduling based on safe deep reinforcement learning. IEEE Trans Smart Grid 11(3):2427–2439. https://doi.org/10.1109/TSG.2019.2955437 CrossRef

22.

Chukwu UC, Mahajan SM (2017) Modeling of V2G net energy injection into the grid. In: 6th International conference on clean electrical power: renewable energy resources impact, ICCEP 2017, vol 29117, pp 437–440. https://doi.org/10.1109/ICCEP.2017.8004724

23.

Sutton Richard S, Barto Andrew G (2018) Reinforcement learning: an introduction, 2nd edn. The MIT Press, Cambridge MATH

24.

Lange S, Gabel T, Riedmiller M (2012). In: Wiering M, van Otterlo M (eds) Batch reinforcement learning. Springer, Berlin, pp 45–73. https://doi.org/10.1007/978-3-642-27645-3_2

25.

Li Z, Chu T, Kalabic U (2019) Dynamics-enabled safe deep reinforcement learning: case study on active suspension control. In: 3rd IEEE conference on control technology and applications, CCTA 2019, pp 585–591. https://doi.org/10.1109/CCTA.2019.8920696

26.

Gan L, Topcu U, Low SH (2012) Stochastic distributed protocol for electric vehicle charging with discrete charging rate. In: IEEE power and energy society general meeting, pp 1–8. https://doi.org/10.1109/PESGM.2012.6344847

27.

Ye Y, Qiu D, Ward J, Abram M (2020) Model-free real-time autonomous energy management for a residential multi-carrier energy system: a deep reinforcement learning approach. In: IJCAI international joint conference on artificial intelligence 2021-January, pp 339–346. https://doi.org/10.24963/ijcai.2020/48

28.

Kiaee F (2020) Integration of electric vehicles in smart grid using deep reinforcement learning. In: 11th International conference on information and knowledge technology, IKT 2020, pp 40–44. https://doi.org/10.1109/IKT51791.2020.9345625

29.

Falvo MC, Sbordone D, Bayram IS, Devetsikiotis M (2014) EV charging stations and modes: International standards. In: International symposium on power electronics, electrical drives, automation and motion, SPEEDAM 2014, pp 1134–1139. https://doi.org/10.1109/SPEEDAM.2014.6872107

30.

EURELECTRIC: dynamic pricing in electricity supply (February), 16 (2017)

31.

Ortega-Vazquez MA (2014) Optimal scheduling of electric vehicle charging and vehicle-to-grid services at household level including battery degradation and price uncertainty. IET Gener Transm Distrib 8(6):1007–1016. https://doi.org/10.1049/iet-gtd.2013.0624 CrossRef

32.

Harm van S, Hado van H, Whiteson S, Wiering M (2009) A theoretical and empirical analysis of expected sarsa. In: Proceedings of IEEE symposium on adaptive dynamic programming and reinforcement learning, ADPRL 2009, pp 177–184. https://doi.org/10.1109/ADPRL.2009.4927542

33.

Tuntispot-hinnat Sähköpörssissä. https://www.vattenfall.fi/sahkosopimukset/porssisahko/tuntispot-hinnat-sahkoporssissa/. Accessed 13 July 2020

Title: Reinforcement learning-based demand-side management by smart charging of electric vehicles
Authors: Melik Bugra Ozcelik
Mert Kesici
Necati Aksoy
Istemihan Genc
Publication date: 06-07-2022
Publisher: Springer Berlin Heidelberg
Published in: Electrical Engineering / Issue 6/2022
Print ISSN: 0948-7921
Electronic ISSN: 1432-0487
DOI: https://doi.org/10.1007/s00202-022-01597-2

Springer Professional

Hint

Swipe to navigate through the articles of this issue

Abstract

Please log in to get access to this content

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 6/2022

Torque ripple reduction in three-level five-phase inverter-fed five-phase induction motor

Predictive sliding surface control of squirrel cage induction motor fed by a voltage source inverter: experimental validation and analyses

The effect of stator slot number and pole number on motor performance in double-sided axial flux induction motors for electric vehicles

Three-phase inverters open-circuit faults diagnosis using an enhanced variational mode decomposition, wavelet packet analysis, and scalar indicators

Tariff rationalization for reactive power and harmonics regulation in low voltage distribution networks

Analysis of DC low energy discharges emitted light for transformer oil state assessment