# An Online Energy Management Control for Hybrid Electric Vehicles Based on Neuro-Dynamic Programming

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Preparation of Knowledge about the NDP EMS

#### 2.1. Hybrid Powertrain

#### 2.2. Optimal Control Problem

#### 2.3. Wavelet Transformation Theory

#### 2.4. Common Wavelet Neural Network

#### 2.5. Multi-Resolution Analysis of Functions

- (1)
- Causality and consistency.$$\cdots \subset {V}_{m+1}\subset {V}_{m}\subset {V}_{m-1}\subset \cdots ,\text{}m\in Z$$
- (2)
- Scaling regularity. When we approximate a function $f(x)\in {L}^{2}(R)$ with ${f}_{m}(x)={A}_{m}f(x)$ at the resolution ${2}^{m}$, where ${A}_{m}$ is a projection of $f(x)$ in subspace ${V}_{m}$. Then the following equation will be satisfied:$${f}_{m}(x)={A}_{m}f(x)\in {V}_{m}\iff {f}_{m-1}(x)={A}_{m-1}f(2x)\in {V}_{m-1},\text{}m\in Z.$$
- (3)
- Gradual completeness.$$\underset{m=-\infty}{\overset{m=+\infty}{\cap}}{V}_{m}=\left\{0\right\},\text{}\underset{m=-\infty}{\overset{m=+\infty}{\cup}}{V}_{m}={L}^{2}(R)$$
- (4)
- Orthogonal basis existence.$${V}_{m}=\overline{linear\text{}span\left\{{\varphi}_{mk},k\in Z\right\}},$$

#### 2.6. Multi-Resolution Wavelet Neural Network

## 3. Proposed NDP for Parallel HEVs

#### 3.1. Critic Network Description

#### 3.2. Action Network Description

#### 3.3. NDP Implementation Procedure

- Select an appropriate structure for the critic network and action network separately, which means determining the number of hidden layers. In this design, the number of hidden layers of the critic network and action network is 30 and 11, respectively.
- Initialize the weights and parameters of both networks according to a rule-based EMS. Initialize the output of the critic network, the maximum training error of critic network ${E}_{c}$, the maximum of the norm for weights of action network variation $\epsilon $, and maximum times for training both networks, ${N}_{c}$ and ${N}_{a}$. In this research, the maximum training times for the action network and critic network are all 50.
- Set the whole iteration time $i=0$ and $L(x(0),u(0))=0$.
- Set $i=i+1$. Calculate the current control vector $u(i)$.
- Set the iteration time of critic network to $m=0$.
- Set $m=m+1$. Calculate the error of critic network ${e}_{c}^{m}(i)$ and ${E}_{c}^{m}(i)$ according to Equations (32) and (33). Update the weights and parameters of the critic network according to Equations (13) and (14);
- When $m\ge {N}_{c}$ or ${E}_{c}^{m}\le {E}_{c}$, save the weights and parameters of the critic network as the weights and parameters at time step $k$, and proceed to step 8. Otherwise, return to step 6.
- Set the iteration time of action network to $m=0$.
- Set $m=m+1$. Calculate the error of action network ${e}_{a}^{m}(i)$ and ${E}_{a}^{m}(i)$ according to Equations (34) and (35). Update the weights and parameters of the action network according to Equations (13) and (14);
- When $m\ge {N}_{a}$ or ${E}_{a}^{m}\le {E}_{a}$, save the weights and parameters of the critic network as the weights and parameters at time step $k$, and proceed to step 11. Otherwise, return to step 9.
- When the norm of weights for action network variation is less than $\epsilon $, save the weights of both networks as the weights at the current time step; otherwise, return to step 4.

## 4. Results and Discussion

## 5. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## References

- Chen, J.; Xu, C.; Wu, C.; Xu, W. Adaptive fuzzy logic control of fuel-cell-battery hybrid systems for electric vehicles. IEEE Trans. Ind. Inform.
**2018**, 14, 292–300. [Google Scholar] [CrossRef] - Delprat, S.; Lauber, J.; Guerra, T.M.; Rimaux, J. Control of a parallel hybrid powertrain: Optimal control. IEEE Trans. Veh. Technol.
**2004**, 53, 872–881. [Google Scholar] [CrossRef] - Kim, N.; Cha, S.; Peng, H. Optimal control of hybrid electric vehicles based on Pontryagin’s minimum principle. IEEE Trans. Control Syst. Technol.
**2011**, 19, 1279–1286. [Google Scholar] - Perez, L.V.; Bossio, G.R.; Moitre, D.; Garcia, G.O. Optimization of power management in an hybrid electric vehicle using dynamic programming. Math. Comput. Simul.
**2006**, 73, 244–254. [Google Scholar] [CrossRef] - Stephan, U.; Nikolce, M.; Conny, T.; Bernard, B. Optimal energy management and velocity control of hybrid electric vehicles. IEEE Trans. Veh. Technol.
**2018**, 67, 327–337. [Google Scholar] - Qin, F.; Xu, G.; Hu, Y.; Xu, K.; Li, W. Stochastic optimal control of parallel hybrid electric vehicles. Energies
**2017**, 10, 214. [Google Scholar] [CrossRef] - Munoz, P.M.; Correa, G.; Gaudiano, M.E.; Fernandez, D. Energy management control design for fuel cell hybrid electric vehicles using neural networks. Int. J. Hydrog. Energy
**2017**, 42, 28932–28944. [Google Scholar] [CrossRef] - Zhang, Q.; Deng, W.; Li, G. Stochastic control of predictive power management for battery/supercapacitor hybrid energy storage systems of electric vehicles. IEEE Trans. Ind. Inform.
**2018**. [Google Scholar] [CrossRef] - Zhang, F.; Xi, J.; Langari, R. Real-time energy management strategy based on velocity forecasts using V2V and V2I communications. IEEE Trans. Control Syst.
**2017**, 18, 416–430. [Google Scholar] [CrossRef] - Song, C.; Lee, H.; Kang, C.; Lee, W.; Kim, Y.B.; Cha, S.W. Traffic speed prediction under weekday using convolutional neural networks concepts. In Proceedings of the IEEE Intelligent Vehicles Symposium, Redondo Beach, CA, USA, 11–14 June 2017; IEEE: New York, NY, USA, 2017; pp. 1293–1298. [Google Scholar]
- Tian, H.; Lu, Z.; Huang, Y.; Tian, G. Adaptive fuzzy logic energy management strategy based on reasonable SOC reference curve for online control of Plug-in Hybrid electric city bus. IEEE Trans. Intell. Transp. Syst.
**2018**. [Google Scholar] [CrossRef] - Qi, X.; Luo, Y.; Wu, G.; Boriboonsomsin, K.; Brath, M.J. Deep reinforcement learning-based vehicle energy efficiency autonomous learning system. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Redondo Beach, CA, USA, 11–14 June 2017. [Google Scholar]
- Hu, Y.; Li, W.; Xu, K.; Zahid, T.; Qin, F.; Li, C. Energy management strategy for a hybrid electric vehicle based on deep reinforcement learning. Appl. Sci.
**2018**, 8, 187. [Google Scholar] [CrossRef] - Li, W.; Xu, G.; Xu, Y. Online learning control for hybrid electric vehicle. Chin. J. Mech. Eng.
**2012**, 25, 98–106. [Google Scholar] [CrossRef] - Mallat, S.G. Multiresolution approximations and wavelet orthogonal bases of L
^{2}(R). Trans. Am. Math. Soc.**1989**, 315, 67–87. [Google Scholar] - Mallat, S.G. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell.
**1989**, 11, 674–693. [Google Scholar] [CrossRef] - Moody, J. Fast Learning in Multi-Resolution Hierarchies; research report; Yale University: New Haven, CT, USA, 1989. [Google Scholar]
- Valenzuela, V.V.; Oliveira, H.M. Close expressions for Meyer wavelet and scale function. arXiv, 2015; arXiv:1502.00161. [Google Scholar]

Item | Value |
---|---|

Final drive gear efficiency (%) | 0.9296 |

wheel radius (m) | 0.275 |

Coefficient of aerodynamic drag | 0.25 |

Vehicle frontal area (m^{2}) | 1.9 |

Air density (kg/m^{3}) | 1.2 |

Rolling resistance coefficient | 0.0054 |

Vehicle mass (kg) | 1000 |

EMS | NDP in [14] | Proposed NDP |
---|---|---|

Fuel Economy (mpg) | 81.6 | 86.0 |

SOC range | 0.55–0.7 | 0.54–0.7 |

Engine Eff. (%) | 34.6 | 35.0 |

Motoring Eff. (%) | 91.6 | 86.1 |

Generating Eff. (%) | 92.7 | 99.7 |

EMS | SDP | Proposed NDP |
---|---|---|

Fuel Economy (mpg) | 68.3 | 86.0 |

SOC range | 0.54–0.7 | 0.54–0.7 |

Engine Eff. (%) | 35.7 | 35.0 |

Motoring Eff. (%) | 91.1 | 86.1 |

Generating Eff. (%) | 93.4 | 99.7 |

EMS | SDP | Proposed NDP |
---|---|---|

Fuel Economy (mpg) | 58.2 | 61.8 |

SOC range | 0.54–0.7 | 0.55–0.7 |

Engine Eff. (%) | 36.1 | 33.5 |

Motoring Eff. (%) | 87.0 | 88.6 |

Generating Eff. (%) | 90.5 | 95.2 |

**Table 5.**Control orientation improvement with another optimal method under CYC_1015 + CYC_UDDS + CYC_WVUCITY.

EMS | SDP | Proposed NDP |
---|---|---|

Fuel Economy (mpg) | 62.1 | 78.0 |

SOC range | 0.54–0.7 | 0.53–0.7 |

Engine Eff. (%) | 36.3 | 36.1 |

Motoring Eff. (%) | 86.1 | 83.7 |

Generating Eff. (%) | 91.8 | 95.5 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Qin, F.; Li, W.; Hu, Y.; Xu, G.
An Online Energy Management Control for Hybrid Electric Vehicles Based on Neuro-Dynamic Programming. *Algorithms* **2018**, *11*, 33.
https://doi.org/10.3390/a11030033

**AMA Style**

Qin F, Li W, Hu Y, Xu G.
An Online Energy Management Control for Hybrid Electric Vehicles Based on Neuro-Dynamic Programming. *Algorithms*. 2018; 11(3):33.
https://doi.org/10.3390/a11030033

**Chicago/Turabian Style**

Qin, Feiyan, Weimin Li, Yue Hu, and Guoqing Xu.
2018. "An Online Energy Management Control for Hybrid Electric Vehicles Based on Neuro-Dynamic Programming" *Algorithms* 11, no. 3: 33.
https://doi.org/10.3390/a11030033