# Utilizing B-Spline Curves and Neural Networks for Vehicle Trajectory Prediction in an Inverse Reinforcement Learning Framework

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

#### 2.1. Interaction-Aware Models

#### 2.2. Trajectory Prediction Using IRL

#### 2.3. Trajectory Prediction for Intersection Safety

## 3. Materials and Methods

#### 3.1. Data Description

#### Data Cleaning and Organization

#### 3.2. Methodology

#### 3.2.1. B-Splines

#### 3.2.2. Conditional Variational Autoencoders

#### 3.2.3. Inverse Reinforcement Learning

#### The Reinforcement Learning (RL) Problem

#### The Inverse Reinforcement Learning (IRL) Problem

## 4. Results and Discussion

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Mohamed, M.G.; Saunier, N. Motion Prediction Methods for Surrogate Safety Analysis. Transp. Res. Rec.
**2013**, 2386, 168–178. [Google Scholar] [CrossRef] - Wolterman, M. Infrastructure-Based Collision Warning Using Artificial Intelligence. U.S. Patent 7,317,406 B2, 8 January 2008. [Google Scholar]
- Lefèvre, S.; Vasquez, D.; Laugier, C. A Survey on Motion Prediction and Risk Assessment for Intelligent Vehicles. ROBOMECH J.
**2014**, 1, 1. [Google Scholar] [CrossRef] [Green Version] - Schreier, M.; Willert, V.; Adamy, J. An Integrated Approach to Maneuver-Based Trajectory Prediction and Criticality Assessment in Arbitrary Road Environments. IEEE Trans. Intell. Transport. Syst.
**2016**, 17, 2751–2766. [Google Scholar] [CrossRef] - Hermes, C.; Wohler, C.; Schenk, K.; Kummert, F. Long-Term Vehicle Motion Prediction. In Proceedings of the 2009 IEEE Intelligent Vehicles Symposium, Xi’an, China, 3–5 June 2009; pp. 652–657. [Google Scholar]
- Djuric, N.; Radosavljevic, V.; Cui, H.; Nguyen, T.; Chou, F.-C.; Lin, T.-H.; Schneider, J. Motion Prediction of Traffic Actors for Autonomous Driving Using Deep Convolutional Networks. arXiv
**2018**, arXiv:1808.05819. [Google Scholar] - Vasquez, D.; Fraichard, T.; Laugier, C. Growing Hidden Markov Models: An Incremental Tool for Learning and Predicting Human and Vehicle Motion. Int. J. Robot. Res.
**2009**, 28, 1486–1506. [Google Scholar] [CrossRef] [Green Version] - Gettman, D.; Head, L. Surrogate Safety Measures from Traffic Simulation Models. Transp. Res. Rec.
**2003**, 1840, 104–115. [Google Scholar] [CrossRef] [Green Version] - Wiest, J.; Höffken, M.; Kreßel, U.; Dietmayer, K. Probabilistic Trajectory Prediction with Gaussian Mixture Models. In Proceedings of the 2012 IEEE Intelligent Vehicles Symposium, Madrid, Spain, 3–7 June 2012; pp. 141–146. [Google Scholar]
- Ammoun, S.; Nashashibi, F. Real Time Trajectory Prediction for Collision Risk Estimation between Vehicles. In Proceedings of the 2009 IEEE 5th International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania, 27–29 August 2009; pp. 417–422. [Google Scholar]
- Pyykönen, P.; Molinier, M.; Klunder, G.A. Traffic Monitoring and Modeling for Intersection Safety. In Proceedings of the 2010 IEEE 6th International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania, 26–28 August 2010; pp. 401–408. [Google Scholar]
- Tan, C.; Zhou, N.; Wang, F.; Tang, K.; Ji, Y. Real-Time Prediction of Vehicle Trajectories for Proactively Identifying Risky Driving Behaviors at High-Speed Intersections. Transp. Res. Rec.
**2018**, 2672, 233–244. [Google Scholar] [CrossRef] - Wang, Y.; Wenjuan, E.; Tian, D.; Lu, G.; Yu, G.; Wang, Y. Vehicle Collision Warning System and Collision Detection Algorithm Based on Vehicle Infrastructure Integration. In Proceedings of the 7th Advanced Forum on Transportation of China (AFTC 2011), Beijing, China, 22 October 2011; pp. 216–220. [Google Scholar]
- Fu, Y.; Li, C.; Luan, T.H.; Zhang, Y.; Mao, G. Infrastructure-Cooperative Algorithm for Effective Intersection Collision Avoidance. Transp. Res. Part C Emerg. Technol.
**2018**, 89, 188–204. [Google Scholar] [CrossRef] - De Campos, G.R.; Runarsson, A.H.; Granum, F.; Falcone, P.; Alenljung, K. Collision Avoidance at Intersections: A Probabilistic Threat-Assessment and Decision-Making System for Safety Interventions. In Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China, 8–11 October 2014; pp. 649–654. [Google Scholar]
- Schendzielorz, T.; Mathias, P.; Busch, F. Infrastructure-Based Vehicle Maneuver Estimation at Urban Intersections. In Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, The Netherlands, 6–9 October 2013; pp. 1442–1447. [Google Scholar]
- Weidl, G.; Breuel, G.; Singhal, V. Collision Risk Prediction and Warning at Road Intersections Using an Object Oriented Bayesian Network. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications—AutomotiveUI ’13, Eindhoven, The Netherlands, 28–30 October 2013; ACM Press: New York, NY, USA, 2013; pp. 270–277. [Google Scholar]
- Wang, L.; Zhang, L.; Zhang, W.-B.; Zhou, K. Red Light Running Prediction for Dynamic All-Red Extension at Signalized Intersection. In Proceedings of the 2009 12th International IEEE Conference on Intelligent Transportation Systems, St. Louis, MO, USA, 4–7 October 2009; pp. 1–5. [Google Scholar]
- Wang, P.; Chan, C.-Y. Vehicle Collision Prediction at Intersections Based on Comparison of Minimal Distance between Vehicles and Dynamic Thresholds. IET Intell. Transp. Syst.
**2017**, 11, 676–684. [Google Scholar] [CrossRef] - Tran, Q.; Firl, J. Online Maneuver Recognition and Multimodal Trajectory Prediction for Intersection Assistance Using Non-Parametric Regression. In Proceedings of the 2014 IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, USA, 8–11 June 2014; pp. 918–923. [Google Scholar]
- Oliver, N.; Pentland, A.P. Graphical Models for Driver Behavior Recognition in a SmartCar. In Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511), Dearborn, MI, USA, 5 October 2000; pp. 7–12. [Google Scholar]
- Brand, M.; Oliver, N.; Pentland, A. Coupled Hidden Markov Models for Complex Action Recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, PR, USA, 17–19 June 1997; pp. 994–999. [Google Scholar]
- Schaal, S. Learning from Demonstration. In Advances in Neural Information Processing Systems 9; Mozer, M.C., Jordan, M.I., Petsche, T., Eds.; MIT Press: Cambridge, MA, USA, 1997; pp. 1040–1046. [Google Scholar]
- Jazayeri, M.S.; Jahangiri, A.; Machiani, S.G. Predicting Vehicle Trajectories at Intersections Using Advanced Machine Learning Techniques; SAFE-D, Safety Through Disruption National University Transportation Center: San Diego, CA, USA, 2021. [Google Scholar]
- Russell, S.J. Learning Agents for Uncertain Environments. In Proceedings of the 11th Annual Conference on Computational Learning Theroy, Madison, WI, USA, 24–26 July 1998; Volume 98, pp. 101–103. [Google Scholar]
- Sharifzadeh, S.; Chiotellis, I.; Triebel, R.; Cremers, D. Learning to Drive Using Inverse Reinforcement Learning and Deep Q-Networks. arXiv
**2016**, arXiv:1612.03653. [Google Scholar] - Kuderer, M.; Gulati, S.; Burgard, W. Learning Driving Styles for Autonomous Vehicles from Demonstration. In Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, 26–30 May 2015; pp. 2641–2646. [Google Scholar]
- Lee, N.; Choi, W.; Vernaza, P.; Choy, C.B.; Torr, P.H.S.; Chandraker, M. DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2165–2174. [Google Scholar]
- Kovvali, V.; Alexiadis, V.; Zhang, L. Video-Based Vehicle Trajectory Data Collection (No. 07-0528). In Proceedings of the Transportation Research Board 86th Annual Meeting, Washington, DC, USA, 21–25 January 2007. [Google Scholar]
- Lin, C.-F.; Ulsoy, A.G.; LeBlanc, D.J. Vehicle Dynamics and External Disturbance Estimation for Vehicle Path Prediction. IEEE Trans. Control Syst. Technol.
**2000**, 8, 508–518. [Google Scholar] [CrossRef] - Barth, A.; Franke, U. Where Will the Oncoming Vehicle Be the next Second? In Proceedings of the 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands, 4–6 June 2008; pp. 1068–1073. [Google Scholar]
- Veeraraghavan, H.; Papanikolopoulos, N.; Schrater, P. Deterministic Sampling-Based Switching Kalman Filtering for Vehicle Tracking. In Proceedings of the 2006 IEEE Intelligent Transportation Systems Conference, Toronto, ON, Canada, 17–20 September 2006; pp. 1340–1345. [Google Scholar]
- Shao, Q.B.; Guan, H.; Jia, X. Vehicle Trajectory Prediction Based on Road Recognition. Appl. Mech. Mater.
**2014**, 599, 760–766. [Google Scholar] [CrossRef] - Käfer, E.; Hermes, C.; Wöhler, C.; Ritter, H.; Kummert, F. Recognition of Situation Classes at Road Intersections. In Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, 3–7 May 2010. [Google Scholar]
- Liebner, M.; Baumann, M.; Klanner, F.; Stiller, C. Driver Intent Inference at Urban Intersections Using the Intelligent Driver Model. In Proceedings of the 2012 IEEE Intelligent Vehicles Symposium, Madrid, Spain, 3–7 June 2012; pp. 1162–1167. [Google Scholar]
- Agamennoni, G.; Nieto, J.I.; Nebot, E.M. Estimation of Multivehicle Dynamics by Considering Contextual Information. IEEE Trans. Robot.
**2012**, 28, 855–870. [Google Scholar] [CrossRef] - Gindele, T.; Brechtel, S.; Dillmann, R. A Probabilistic Model for Estimating Driver Behaviors and Vehicle Trajectories in Traffic Environments. In Proceedings of the 13th International IEEE Conference on Intelligent Transportation Systems, Funchal, Portugal, 19–22 September 2010; pp. 1625–1631. [Google Scholar]
- Wiest, J. Statistical Long-Term Motion Prediction. Ph.D. Thesis, Universität Ulm, Institut für Mess-, Regel- und Mikrotechnik, Ulm, Germany, 2016. [Google Scholar]
- Agamennoni, G.; Nieto, J.I.; Nebot, E.M. A Bayesian Approach for Driving Behavior Inference. In Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany, 5–9 June 2011; pp. 595–600. [Google Scholar]
- Lefevre, S.; Laugier, C.; Ibanez-Guzman, J. Risk Assessment at Road Intersections: Comparing Intention and Expectation. In Proceedings of the 2012 IEEE Intelligent Vehicles Symposium, Madrid, Spain, 3–7 June 2012; pp. 165–171. [Google Scholar]
- Kuefler, A.; Morton, J.; Wheeler, T.; Kochenderfer, M. Imitating Driver Behavior with Generative Adversarial Networks. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017; pp. 204–211. [Google Scholar]
- Hubmann, C.; Schulz, J.; Becker, M.; Althoff, D.; Stiller, C. Automated Driving in Uncertain Environments: Planning with Interaction and Uncertain Maneuver Prediction. IEEE Trans. Intell. Veh.
**2018**, 3, 5–17. [Google Scholar] [CrossRef] - Liu, W.; Kim, S.-W.; Pendleton, S.; Ang, M.H. Situation-Aware Decision Making for Autonomous Driving on Urban Road Using Online POMDP. In Proceedings of the 2015 IEEE Intelligent Vehicles Symposium (IV), Seoul, Korea, 28 June–1 July 2015; pp. 1126–1133. [Google Scholar]
- Shirazi, M.S.; Morris, B.T. Trajectory Prediction of Vehicles Turning at Intersections Using Deep Neural Networks. Mach. Vis. Appl.
**2019**, 30, 1097–1109. [Google Scholar] - Deo, N.; Trivedi, M.M. Convolutional Social Pooling for Vehicle Trajectory Prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018; pp. 1468–1476. [Google Scholar]
- Sarkar, A.; Czarnecki, K.; Angus, M.; Li, C.; Waslander, S. Trajectory Prediction of Traffic Agents at Urban Intersections through Learned Interactions. In Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, 16–19 October 2017; pp. 1–8. [Google Scholar]
- Sun, L.; Zhan, W.; Tomizuka, M. Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 2111–2117. [Google Scholar]
- Sun, R.; Hu, S.; Zhao, H.; Moze, M.; Aioun, F.; Guillemard, F. Human-like Highway Trajectory Modeling Based on Inverse Reinforcement Learning. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 27–30 October 2019; pp. 1482–1489. [Google Scholar]
- Choi, D.; Min, K.; Choi, J. Regularizing Neural Networks for Future Trajectory Prediction via Inverse Reinforcement Learning Framework. arXiv
**2019**, arXiv:1907.04525. [Google Scholar] - Hjaltason, B. Predicting Vehicle Trajectories with Inverse Reinforcement Learning; KTH Royal Institute of Technology: Stockholm, Sweden, 2019. [Google Scholar]
- Sohn, K.; Lee, H.; Yan, X. Learning Structured Output Representation Using Deep Conditional Generative Models. Adv. Neural Inf. Processing Syst.
**2015**, 28, 3483–3491. [Google Scholar] - Kingma, D.P.; Welling, M. Auto-Encoding Variational Bayes. arXiv
**2014**, arXiv:1312.6114. [Google Scholar] - Ziebart, B.D.; Maas, A.; Bagnell, J.A.; Dey, A.K. Maximum Entropy Inverse Reinforcement Learning. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008), Chicago, IL, USA, 13–17 July 2008; Volume 8, pp. 1433–1438. [Google Scholar]
- Finn, C.; Levine, S.; Abbeel, P. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 20–22 June 2016; p. 10. [Google Scholar]
- Ren, P.; Xiao, Y.; Chang, X.; Huang, P.; Li, Z.; Chen, X.; Wang, X. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions. ACM Comput. Surv.
**2022**, 54, 1–34. [Google Scholar] [CrossRef]

**Figure 1.**An overview of the method, reprinted from [24].

**Figure 2.**The trajectory of a left turn and the predictions of our method, reprinted from [24].

Study | Environment | Methods | Predictors |
---|---|---|---|

[26] | Highway | IRL and Deep Q-Nets | Surroundings |

[27] | Highway | IRL | Surroundings |

[47] | Road Segment | Hierarchical IRL | Surroundings |

[28] | Intersection and Road Segment | Recurrent Neural Networks and IRL | Previous Trajectory and Surroundings |

[48] | Highway | IRL | Surroundings |

[50] | Highway | IRL | Previous Trajectory and Surroundings |

Study | Predictors (Detail) | Data Collection Sensors | Monitoring Period | Prediction Horizon | Evaluation Metric | Tested Applications | Interaction Type | Movement Type |
---|---|---|---|---|---|---|---|---|

[12] | Position, velocity, distance to preceding vehicle, speed difference from preceding vehicle | Video camera | 1 s | 12 s | RMSE of difference between predicted and actual trajectory | Detect red light running, abrupt stops, aggressive passes, speeding passes, and aggressive following | Vehicle, vehicle-vehicle | all |

[44] | Vehicle position over a number of preceding frames | Video camera | 1/3 of each trajectory | 2 s | Turning prediction accuracy | Early prediction of turning movements | Vehicle-vehicle, vehicle-pedestrian | all |

[10] | Vehicle position, velocity, and acceleration | GPS | Up to the prediction point | 10 s | No quantitative evaluation | Collision detection and risk assessment | Vehicle-vehicle | all |

[13] | Vehicle position and velocity | DGPS | Not Specified | Not Specified | No Quantitative Evaluation | Collision detection and warning | Vehicle-vehicle | all |

[46] | Vehicle position, velocity, and previous trajectory + surroundings | Video camera | Not specified | 0–3 s | RMSE of difference between predicted and actual trajectory | - | Vehicle-vehicle, vehicle-pedestrian | all |

[15] | Vehicle position, speed, acceleration, and yaw | GPS + inertial sensors | Not specified | Not specified | No quantitative Evaluation | Frontal collision prevention/mitigation | vehicle-vehicle | Frontal collisions caused by any movement |

[14] | Vehicle position, velocity, acceleration, distance traveled, turn signal, road condition | Simulation | Not specified | Not specified | TPR, FPR, and FNR for collision prediction and Collision avoidance success | Collision avoidance and warning | Vehicle-vehicle | All movements |

[11] | Vehicle position, velocity | Video camera | Prediction performed at every time step | Not specified | No quantitative Evaluation | Collision detection | Vehicle-vehicle, vehicle-pedestrian | All movements |

[16] | Vehicle position, velocity, acceleration | Roadside sensors, on board GPS | Not specified | Maximum of 10 s | Levels of accident mitigation | Collision mitigation | Vehicle-vehicle, vehicle-cyclist, vehicle-pedestrian | Turns and red light running |

[18] | Vehicle position, velocity, and acceleration | Video camera | Not specified | Not specified | Simulated SOC curve | Red light running prediction | - | Red light running |

[17] | Vehicle position, velocity, acceleration | Intersection mounted cameras and laser sensors + on board sensors | Not specified | 2 s | No quantitative evaluation | Collision risk prediction | Vehicle-vehicle, vehicle-pedestrian, vehicle-cyclist | All movements |

[19] | Vehicle position, velocity, acceleration | Not specified | Not specified | 3 s | False positive + false negative | Collision prediction and warning | Vehicle-vehicle | All movements |

Our work | Vehicle position, velocity, acceleration + surroundings | Video camera | 2 s | 3 s | RMSE | - | Vehicle-vehicle | All movements |

Intersection | Total Rows | Total Trajectories | Right Turns | Left Turns | Through | Number of Autos | Number of Trucks | Number of Motorcycles |
---|---|---|---|---|---|---|---|---|

2 | 574,398 | 2210 | 157 | 616 | 1437 | 2144 | 62 | 4 |

3 | 193,028 | 1973 | 24 | 82 | 1867 | 1915 | 54 | 4 |

4 | 218,049 | 1980 | 214 | 619 | 1147 | 1917 | 59 | 4 |

Method | Avg. RMSE (m) |
---|---|

Baseline (Kalman Filter) | 5.1 |

Neural Network | 4.6 |

Neural Network + IRL Ranking | 4.1 |

[5] ^{†} | 5 |

[12] ^{†} | 5.02 |

^{†}On different datasets.

Movement Type | Avg. RMSE (m) without IRL Scoring | Avg RMSE (m) With IRL Scoring |
---|---|---|

Through | 2.9 | 2.6 |

Right | 14.7 | 12.8 |

Left | 13.1 | 11.3 |

Prediction Horizon (s) | Avg. RMSE without IRL Scoring | Avg RMSE with IRL Scoring |
---|---|---|

1 | 0.7 | 0.6 |

2 | 2.1 | 1.9 |

3 | 4.6 | 4.1 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jazayeri, M.S.; Jahangiri, A.
Utilizing B-Spline Curves and Neural Networks for Vehicle Trajectory Prediction in an Inverse Reinforcement Learning Framework. *J. Sens. Actuator Netw.* **2022**, *11*, 14.
https://doi.org/10.3390/jsan11010014

**AMA Style**

Jazayeri MS, Jahangiri A.
Utilizing B-Spline Curves and Neural Networks for Vehicle Trajectory Prediction in an Inverse Reinforcement Learning Framework. *Journal of Sensor and Actuator Networks*. 2022; 11(1):14.
https://doi.org/10.3390/jsan11010014

**Chicago/Turabian Style**

Jazayeri, Mohammad Sadegh, and Arash Jahangiri.
2022. "Utilizing B-Spline Curves and Neural Networks for Vehicle Trajectory Prediction in an Inverse Reinforcement Learning Framework" *Journal of Sensor and Actuator Networks* 11, no. 1: 14.
https://doi.org/10.3390/jsan11010014