AutoDRIVE: A Comprehensive, Flexible and Integrated Digital Twin Ecosystem for Autonomous Driving Research & Education

Abstract

Prototyping and validating hardware–software components, sub-systems and systems within the intelligent transportation system-of-systems framework requires a modular yet flexible and open-access ecosystem. This work presents our attempt to develop such a comprehensive research and education ecosystem, called AutoDRIVE, for synergistically prototyping, simulating and deploying cyber-physical solutions pertaining to autonomous driving as well as smart city management. AutoDRIVE features both software as well as hardware-in-the-loop testing interfaces with openly accessible scaled vehicle and infrastructure components. The ecosystem is compatible with a variety of development frameworks, and supports both single- and multi-agent paradigms through local as well as distributed computing. Most critically, AutoDRIVE is intended to be modularly expandable to explore emergent technologies, and this work highlights various complementary features and capabilities of the proposed ecosystem by demonstrating four such deployment use-cases: (i) autonomous parking using probabilistic robotics approach for mapping, localization, path-planning and control; (ii) behavioral cloning using computer vision and deep imitation learning; (iii) intersection traversal using vehicle-to-vehicle communication and deep reinforcement learning; and (iv) smart city management using vehicle-to-infrastructure communication and internet-of-things.

1. Introduction

Advancing the field of connected autonomous vehicles (CAVs) [1] requires scientific and technological research in conjunction with comprehensive education of methods and tools to overcome existing challenges and prepare the next generation of practitioners.

Inasmuch as meaningful verification and validation (V&V) efforts demand end-to-end stress testing across scales at component/sub-system/system levels, there is a great incentive for exploring the creation and exploitation of varying grades of virtual (simulation-based) and physical (hardware-in-the-loop) testing platforms to alleviate the monetary, spatial, temporal and safety constraints associated with rapid-prototyping of CAV solutions. In a research setting, such platforms can accelerate the process of designing experiments, recording datasets as well as re-iteratively prototyping and validating autonomy solutions. In an educational setting, such platforms can aid in designing interactive demonstrations, hands-on assignments, projects, and competitions.

However, existing platforms for this purpose are observed to limit the throughput of developing and validating connected autonomy solutions. Firstly, most of these platforms lack the integrity required to promote hardware-software co-development; some only offer software simulation tools (e.g., [2,3,4]), while others only provide scaled physical vehicles (e.g., [5,6,7,8]) to test autonomy algorithms. Such isolated platforms not only decelerate the prototyping phase due to compatibility issues, but also adversely affect the validation phase involving simulation to real-world (sim2real) deployments. Secondly, most of these platforms focus specifically on vehicles rather than a holistic intelligent transportation ecosystem involving infrastructure, traffic elements and peer agents, which limits their applications. Thirdly, some of these platforms (e.g., [9,10]) are domain-specific with limited sensing modalities, stringent design requirements and/or fixed development frameworks; some (e.g., [11]) even lack a high-level computation unit and are merely teleoperated from a remote server to execute the intended mission.

This work does not propose “yet another” research and education platform targeting selective aspects of autonomous driving technology. AutoDRIVE (refer Figure 1) aims to provide a cyber-physical ecosystem that is:

• Comprehensive: The ecosystem offers a scaled car-like vehicle with abundant sensors, which supports single- as well as multi-agent algorithms with or without vehicle-to-vehicle (V2V) communication. It also provides a modular infrastructure development kit comprising various environment modules, traffic elements and surveillance elements, which supports internet-of-things (IoT) and vehicle-to-infrastructure (V2I) communication. On the software front, the ecosystem hosts a high-fidelity simulator and supports the development of autonomous driving as well as smart city solutions.
• Flexible: The ecosystem offers modular hardware components, a convenient high-fidelity simulator, and an extensive software development support, which enables the end-users to flexibly prototype and validate their autonomy solutions right out of the box. Additionally, the completely open-hardware, open-software architecture of the ecosystem allows users to adapt any of the existing hardware (including the design of the vehicle as well as the infrastructure modules) and/or software (including the codebase of the development framework as well as the simulator) to better fit their use-cases.
• Integrated: The ecosystem hosts a tightly coupled trio, comprising AutoDRIVE Devkit (to flexibly develop connected autonomy solutions), AutoDRIVE Simulator (to virtually prototype and test them under a variety of conditions and edge-cases), and AutoDRIVE Testbed (to deploy and validate them in controlled real-world settings). The harmony among these three platforms not only enhances the hardware–software co-development of autonomy solutions, but also helps to seamlessly bridge the gap between software simulation and hardware deployment for the verification and validation of these safety-critical systems.

This work also describes sample use-cases of four emergent applications in the field of CAVs, with each exploiting, and thus exhibiting, distinct features and capabilities of the AutoDRIVE Ecosystem. These include state-of-the-art implementations such as autonomous parking, behavioral cloning and intersection traversal, along with a novel implementation of smart city management.

2. State of the Art

The deployment of full-scale CAV solutions generally requires extensive verification and validation, which poses several challenges, especially in university settings. The time, expense, resources and knowledge of full-scale testing present an interplay with the infrastructural requirements as well as the safety of the personnel and property involved, often hindering research and education progress. Consequently, the past decade has witnessed many university-based deployments exploring the development of scaled autonomous vehicles. As described in

Table 1. Comparative analysis of scaled platforms/ecosystems for autonomy research and education.

Platform/Ecosystem				Cost *	Sensing Modalities							Computational Resources		Actuation Mechanism		Dedicated Simulator		V2X Support		API Support
	Scale	Open Hardware	Open Software		Throttle	Steering	Wheel Encoders	GPS/IPS	IMU	LIDAR	Camera	High-Level	Low-Level	Ackermann Steered	Differetial-Drive/ Skid-Steered		Multi-Agent Support	V2V	V2I	C++	Python	ROS	MATLAB/Simulink	Webapp
AutoDRIVE	1:14	✓	✓	$450	✓	✓	✓	✓	✓	✓	✓	Jetson Nano	Arduino Nano	✓	★	✓	✓	✓	✓	✓	✓	✓	★	✓
MIT Racecar	1:10	★	✓	$2600	✗	✗	✗	✗	✓	✓	✓	Jetson TX2	VESC	✓	✗	Gazebo	★	★	✗	✗	✗	✓	✗	✗
AutoRally	1:5	★	✓	$23,300	✗	✗	✓	✓	✓	✓	✓	Custom	Teensy LC/Arduino Micro	✓	✗	Gazebo	★	★	✗	✗	✗	✓	✗	✗
F1TENTH	1:10	★	✓	$3260	✗	✗	✗	✗	✗	✓	✗	Jetson TX2	VESC 6MkV	✓	✗	RViz/Gazebo	✓	✓	✗	✗	✗	✓	✗	✗
DSV	1:10	★	✓	$1000	✗	✗	✓	✗	✓	✓	✓	ODROID-XU4	Arduino (Mega + Uno)	✓	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗
MuSHR	1:10	★	✓	$930	✗	✗	✗	✗	✗	✗	✓	Jetson Nano	Turnigy SK8-ESC	✓	✗	RViz	✓	✓	✗	✗	✗	✓	✗	✗
HyphaROS RaceCar	1:10	★	✓	$600	✗	✗	✗	✗	✓	✓	✗	ODROID-XU4	RC ESC TBLE-02S	✓	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗
Donkey Car	1:16	★	✓	$370	✗	✗	✗	✗	✗	✗	✓	Raspberry Pi	ESC	✓	✗	Gym	✗	✗	✗	✗	✓	✗	✗	✗
BARC	1:10	★	✓	$1030	✗	✗	✓	✗	✓	✗	✓	ODROID-XU4	Arduino Nano	✓	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗
OCRA	1:43	★	✓	$960	✗	✗	✗	✗	✓	✗	✗	None	ARM Cortex M4 $\mu$C	✓	✗	✗	✓	✗	✗	✓	✗	✗	✓	✗
QCar	1:10	✗	✗	$20,000	✗	✗	✓	✗	✓	✓	✓	Jetson TX2	Proprietary	✓	✗	Simulink	✓	✓	✗	★	★	★	✓	✗
AWS DeepRacer	1:18	✗	✗	$400	✗	✗	✗	✗	✓	★	✓	Proprietary	Proprietary	✓	✗	Gym	✗	✗	✗	✗	✗	✗	✗	✓
Duckietown	N/A	✓	✓	$450	✗	✗	★	✗	★	✗	✓	Raspberry Pi/Jetson Nano	None	✗	✓	Gym	✓	✗	★	✗	✗	✓	✗	✗
TurtleBot3	N/A	✓	✓	$590	✗	✗	✓	✗	✓	✓	★	Raspberry Pi	OpenCR	✗	✓	Gazebo	★	★	✗	✗	✗	✓	✗	✗
Pheeno	N/A	✓	✓	$350	✗	✗	✓	✗	✓	✗	✓	Raspberry Pi	Arduino Pro Mini	✗	✓	✗	✓	✓	✗	✗	✓	★	✗	✗

✓ indicates complete fulfillment; ★ indicates conditional, unsupported or partial fulfillment; and ✗ indicates non-fulfillment. * All cost values are ceiled to the nearest $10.

, such vehicles include the MIT Racecar [5], AutoRally [6], F1TENTH [7], Multi-agent System for non-Holonomic Racing (MuSHR) [8], Optimal RC Racing (ORCA) Project [11], Delft Scaled Vehicle (DSV) [12], and Berkeley Autonomous Race Car (BARC) [13], to name a few. Some of the other community-driven platforms for autonomous driving include HyphaROS RaceCar [9] and Donkey Car [10], both of which are application specific—the former to map-based navigation, and the later to vision-aided imitation learning. Apart from these, commercial products such as QCar [14] by Quanser and DeepRacer [15] by Amazon Web Services (AWS) are now surfacing the market. However, the fact that most of these products are expensive and employ some form of proprietary hardware and/or software components restricts their openness and flexibility to the community, and there are potential issues such as warranty-voids and vendor lock-ins. Some of the other scaled platforms for autonomy research and education include Duckietown [16], TurtleBot3 [17] and Pheeno [18]. However, the differentially driven robots proposed by these platforms/ecosystems fail to fully satisfy the community requirements for a kinodynamically constrained car-like vehicle.

In terms of comprehensiveness, some of these platforms lack diverse sensing modalities, some lack adequate computational power, some lack an Ackermann steering mechanism, and most lack active or passive infrastructural elements. Only a few satisfy the prominent community requirements, but they can prove to be prohibitively expensive for university programs.

In terms of flexibility, most of these platforms, if not all, use commercial-off-the-shelf (COTS) radio-controlled (RC) cars as their base-chassis, which: (a) are quite expensive; (b) may not be available all around the world; (c) limit research on the “mechatronics engineering” front, which is equally important for cyber-physical systems such as CAVs. Additionally, most of these platforms only support a specific software framework, such as a Robot Operating System (ROS) [19], which inherently creates a skillset-dependency for the end-users. Furthermore, providing assets and plugins for pre-packaged simulators such as Gazebo [20] or OpenAI Gym [21] environments offers only so much flexibility to the users in terms of designing and running the simulated scenarios.

In terms of integrity, some of these platforms do not support simulation in any form, some ROS-based ones support kinematic/dynamic simulation using RViz [22] and/or Gazebo, while others offer task-specific Gym environments for imitation/reinforcement learning, none of which is ideal.

3. AutoDRIVE Testbed

AutoDRIVE Testbed is a hardware platform featuring a native scaled vehicle along with a modular and reconfigurable infrastructure development kit for deploying and validating autonomy algorithms in controlled real-world settings. It adopts a completely open-hardware and open-software architecture to push the “systems engineering and integration” front with regard to CAVs.

3.1. Vehicle

AutoDRIVE’s native vehicle, named Nigel (refer Figure 2A,B), offers realistic driving and steering actuation, comprehensive sensor suite, high-performance computational resources, and a standard vehicular lighting system.

3.1.1. Chassis

Nigel is a 1:14 scale model vehicle comprising four modular platforms, each housing distinct components of the vehicle. It adopts rear-wheel drive, the Ackermann steered mechanism (refer Figure 2C), and therefore resembles an actual car in terms of kinodynamic constraints.

3.1.2. Power Electronics

Nigel is powered using an 11.1 V 5200 mAh lithium-polymer (LiPo) battery, whose health is monitored by a voltage checker. A 10 A rated buck converter steps down the voltage to 5 V, which is then routed, via a 3 A rated master switch, to all the electrical sub-systems including a 20 A rated motor driver module.

3.1.3. Sensor Suite

Nigel hosts a comprehensive sensor suite comprising throttle and steering sensors (actuator feedbacks), 1920 CPR incremental encoders (wheel rotation/velocity), a three-axis indoor-positioning system (IPS) using fiducial markers (mm/cm-level accurate small-scale positioning analogous to m-level accurate full-scale GNSS), 9-axis IMU (raw inertial data and calibrated AHRS data using Madgwick/Mahony filter), two 62.2${}^{\circ }$ FOV cameras with 3.04 mm focal length (front and/or rear RGB frames) and a 7–10 Hz, 360${}^{\circ }$ FOV LIDAR with 12 m range and 1${}^{\circ }$ resolution (2D laser scan).

3.1.4. Computation, Communication and Software

Nigel adopts Jetson Nano Developer Kit - B01 for most of its high-level computation (autonomy algorithms), communication (V2V and V2I) and software installation (JetPack SDK, ROS Melodic and AutoDRIVE Devkit). Additionally, it also hosts Arduino Nano (running the vehicle firmware) for acquiring and filtering raw sensor data and controlling actuators/lights.

3.1.5. Actuators

Nigel is provided with two 6 V 160 RPM rated 120:1 DC geared motors to drive its rear wheels, and a 9.4 kgf.cm servo motor to steer its front wheels; the steering actuator is saturated at ±30${}^{\circ }$ w.r.t. zero-steer value. All the actuators are operated at 5 V, which translates to a maximum speed of ∼130 RPM for driving (∼0.267 m/s @ vehicle) and ∼0.19 s/60${}^{\circ }$ for steering (∼0.805 rad/s @ vehicle).

3.1.6. Lights and Indicators

Nigel’s lighting system comprises dual-mode headlights, automated taillights, triple-mode turning indicators and automated reverse indicators.

3.2. Infrastructure

AutoDRIVE offers a modular and reconfigurable infrastructure development kit (refer Figure 3) for rapidly designing and prototyping custom driving scenarios. This kit includes a range of environment modules, traffic elements and surveillance elements, along with several preconfigured maps.

3.2.1. Environment Modules

Environment modules include static layouts and objects meant for rapidly designing custom scenarios. Apart from these, experts may also choose to design scaled real-world or imaginary scenarios using third-party tools, and import them into the AutoDRIVE Ecosystem.

• Terrain Modules: These define off-road segments of the environment. AutoDRIVE currently supports five terrains with tunable physical properties (refer Figure 3A).
• Road Kits: These enable the reconfigurable construction of drivable segments of the environment. AutoDRIVE currently supports 1, 2, 4 and 6 lane road kits, each having 8 different modules (refer Figure 3B).
• Obstruction Modules: These 3D objects define static obstacles within the scene. AutoDRIVE currently supports two such modules (refer Figure 3C).

3.2.2. Traffic Elements

Traffic elements (refer Figure 3D) define traffic laws within a particular driving scenario, thereby governing the traffic flow. AutoDRIVE currently supports modular traffic signs and lights. These modules support IoT and V2I communication technologies, and can be therefore integrated with AutoDRIVE Smart City Manager (SCM).

3.2.3. Surveillance Elements

AutoDRIVE features a surveillance element called AutoDRIVE Eye to view the entire scene from a bird’s-eye view. The said element is also integrated with AutoDRIVE SCM, and, upon calibration of its intrinsic parameters, is capable of estimating vehicle’s 2D pose within the map by detecting and tracking the AprilTag markers attached to each of them; this functionality is illustrated in Figure 3E (notice the roof-mounted camera).

3.2.4. Preconfigured Maps

AutoDRIVE currently offers four preconfigured maps (refer Figure 3F). Parking School is designed specifically for autonomous parking applications, wherein construction boxes define static obstacles and all the available free-space is drivable. The Driving School covers driving over straight roads, driving over curved roads and crossing an intersection. An Intersection School is designed specifically for intersection traversal applications, wherein lane bounds play an important role. Finally, Tiny Town is meant to be a comprehensive driving scenario, which covers each and every infrastructure element currently available in AutoDRIVE.

4. AutoDRIVE Simulator

AutoDRIVE Simulator [23,24] acts as the digital twin of the AutoDRIVE Testbed. It is primarily targeted towards the virtual prototyping of autonomy solutions, either for variability testing or as a part of a recursive simulation-deployment workflow, but can also be used for synthetic data generation.

4.1. Vehicle Dynmaics Simulation

The vehicle is jointly modelled (refer Figure 4A) as a rigid body and as a collection of sprung masses ${}^{i}M$, such that the total mass of the rigid body is $M=\sum {}^{i}M$. The rigid body center of mass $X_{COM}=\frac{\sum {}^{i}M\ast {}^{i}X}{\sum {}^{i}M}$ is what links the said representations, where ${}^{i}X$ are the sprung mass coordinates.

The suspension force acting on each of the sprung masses is ${}^{i}M\ast {}^i\ddot{Z}+{}^{i}B\ast ({}^i\dot{Z}-{}^i\dot{z})+{}^{i}K\ast ({}^{i}Z-{}^{i}z)$; where, ${}^{i}Z$ and ${}^{i}z$ are the displacements of sprung and unsprung masses, and ${}^{i}B$ and ${}^{i}K$ are the damping and spring coefficients of i-th suspension, respectively.

The wheels of the vehicle are also modelled as rigid bodies of mass m that are acted upon by gravitational and suspension forces: ${}^{i}m\ast {}^i\ddot{z}+{}^{i}B\ast ({}^i\dot{z}-{}^i\dot{Z})+{}^{i}K\ast ({}^{i}z-{}^{i}Z)$.

The tire forces are computed based on the friction curve for each tire: $\left\{\begin{array}{c}{}^i{F}_{t_x}=F{(}^i{S}_x)\\ {}^i{F}_{t_y}=F{(}^i{S}_y)\end{array}\right.$; where, ${}^i{S}_x$ and ${}^i{S}_y$ are the longitudinal and lateral slips of i-th tire, respectively. Here, the friction curve is approximated by a two-piece cubic spline $F\left(S\right)=\left\{\begin{array}{c}{f}_0\left(S\right);S_0\le S<S_e\\ f_1\left(S\right);S_e\le S<S_a\end{array}\right.$; where, $f_k\left(S\right)=a_k\ast S^3+b_k\ast S^2+c_k\ast S+d_k$ is a cubic polynomial function. The first segment of the said spline starts at zero $(S_0,F_0)$ and reaches the extremum point $(S_e,F_e)$, while the other segment starts at the extremum point $(S_e,F_e)$ and saturates at the asymptote point $(S_a,F_a)$, as shown in the inset of Figure 4A.

The tire slip is itself affected by various factors including tire stiffness ${}^i{C}_{\alpha }$, steering angle $\delta$, wheel speeds ${}^i\omega$, suspension forces ${}^i{F}_s$, and rigid-body momentum ${}^{i}P$, all of which affect the longitudinal/lateral/both components of the linear velocity of the vehicle. Longitudinal slip ${}^i{S}_x$ of i-th tire is computed by comparing the longitudinal components of surface velocity of i-th wheel (i.e., longitudinal linear velocity of vehicle) $v_x$ with the angular velocity ${}^i\omega$ of i-th wheel: ${}^i{S}_x=\frac{{}^{i}r\ast {}^i\omega -v_x}{v_x}$. Lateral slip ${}^i{S}_y$ of i-th tire is dependent on the direction in which the tire is pointing and the direction in which it is moving, commonly called the slip angle $\alpha$. It is computed by comparing the longitudinal $v_x$ (a.k.a. forward velocity) and lateral $v_y$ (a.k.a. side-slip velocity) components of the vehicle’s linear velocity: ${}^i{S}_y=tan\left(\alpha \right)=\frac{v_y}{\left|v_x\right|}$.

4.2. Sensor Simulation

The simulated vehicle is provided with the same sensing modalities as its real-world counterpart. The throttle ($\tau$) and steering ($\delta$) sensors are simulated through a simple feedback loop.

The incremental encoders are simulated by measuring the rotation of rear wheels (i.e., output shaft of driving actuators): ${}^i{N}_{ticks}={}^{i}PPR\ast {}^{i}GR\ast {}^i{N}_{rev}$; where, ${}^i{N}_{ticks}$ represents ticks measured by the i-th encoder, ${}^{i}PPR$ is the base resolution (pulses per revolution) of i-th encoder, ${}^{i}GR$ is the gear-ratio of i-th motor, and ${}^i{N}_{rev}$ represents the number of revolutions of the output shaft of i-th motor.

The IPS and IMU are simulated based on temporally coherent rigid-body transform updates of the vehicle $\left\{v\right\}$ w.r.t. the world $\left\{w\right\}$: ${}^w{\mathbf{T}}_v=[\begin{array}{c}{\mathbf{R}}_{3\times 3}\\ {\mathbf{0}}_{1\times 3}\end{array}\mid \begin{array}{c}{\mathbf{t}}_{3\times 1}\\ 1\end{array}]\in SE\left(3\right)$. The IPS provides 3-DOF positional coordinates $\{x,y,z\}$ of the vehicle, whereas the IMU provides linear accelerations $\{a_x,a_y,a_z\}$, angular velocities $\{{\omega }_x,{\omega }_y,{\omega }_z\}$ and a 3-DOF orientation of the vehicle as Euler angles $\{{\varphi }_x,{\theta }_y,{\psi }_z\}$ or quaternion $\{q_0,q_1,q_2,q_3\}$.

The LIDAR is simulated using iterative ray-casting raycast{${}^w{\mathbf{T}}_l$, $\overrightarrow{\mathbf{R}}$, $r_{max}$} $\forall \theta \in \left[{\theta }_{min}:{\theta }_{res}:{\theta }_{max}\right]$ at ∼7 Hz update rate, where, ${}^w{\mathbf{T}}_l={}^w{\mathbf{T}}_v\ast {}^v{\mathbf{T}}_l\in SE\left(3\right)$ is the relative transform of LIDAR {l} w.r.t. vehicle {v} w.r.t. world {w}, $\overrightarrow{\mathbf{R}}={\left[r_{max}\ast sin\left(\theta \right)r_{min}\ast cos\left(\theta \right)0\right]}^T$ is the direction vector of each ray-cast R, $r_{min}=$ 0.15 m and $r_{max}=$ 12 m are, respectively, the minimum and maximum linear ranges of LIDAR, ${\theta }_{min}=0^{\circ }$ and ${\theta }_{max}={360}^{\circ }$ are, respectively, the minimum and maximum angular ranges of LIDAR, and ${\theta }_{res}=1^{\circ }$ is the angular resolution of LIDAR. The laser scan ranges are recorded by checking the ray-cast hits and thresholding the minimum linear range of LIDAR: ranges[i]$=\left\{\begin{array}{cc}\mathtt{hit}.\mathtt{dist}\hfill & \mathrm{if}\mathtt{ray}\left[\mathtt{i}\right].\mathtt{hit}\mathrm{and}\mathtt{hit}.\mathtt{dist}\ge r_{min}\hfill \\ \infty \hfill & \mathrm{otherwise}\hfill \end{array}\right.$, where ray.hit is a Boolean flag that checks if a ray-cast hits any colliders in the scene and hit.dist is the Euclidean distance from source of the ray-cast $\{x_{ray},y_{ray},z_{ray}\}$ to the hit-point $\{x_{hit},y_{hit},z_{hit}\}$: $\sqrt{{(x_{hit}-x_{ray})}^2+{(y_{hit}-y_{ray})}^2+{(z_{hit}-z_{ray})}^2}$.

The simulated physical cameras are parameterized by their focal length ($f=$ 3.04 mm), sensor size ($\{s_x,s_y\}=$ {3.68, 2.76} mm), target resolution (default = 720p) and distance from near and far clipping planes ($N=$ 0.01 m and $F=$ 1000 m). The viewport-rendering pipeline for simulated cameras works in three stages. First, the camera view matrix $\mathbf{V}\in SE\left(3\right)$ is computed by taking the relative homogeneous transform of the camera $\left\{c\right\}$ w.r.t. the world $\left\{w\right\}$: $\mathbf{V}=\left[\begin{array}{cccc}{r}_{00}& r_{01}& r_{02}& t_0\\ r_{10}& r_{11}& r_{12}& t_1\\ r_{20}& r_{21}& r_{22}& t_2\\ 0& 0& 0& 1\end{array}\right]$, where $r_{ij}$ and $t_i$ denote rotational and translational components, respectively. Next, the camera projection matrix $\mathbf{P}\in {\mathbb{R}}^{4\times 4}$ is computed by projecting the world coordinates to image space coordinates: $\mathbf{P}=\left[\begin{array}{cccc}\frac{2\ast N}{R-L}& 0& \frac{R+L}{R-L}& 0\\ 0& \frac{2\ast N}{T-B}& \frac{T+B}{T-B}& 0\\ 0& 0& -\frac{F+N}{F-N}& -\frac{2\ast F\ast N}{F-N}\\ 0& 0& -1& 0\end{array}\right]$, where N and F denote distances to near and far clipping planes of the camera, respectively, and L, R, T and B denote the left, right, top and bottom offsets of the sensor, respectively. The camera parameters $\{f,s_x,s_y\}$ are related to the projection matrix terms through the following relations: $f=\frac{2\ast N}{R-L}$, $a=\frac{s_y}{s_x}$, $\frac{f}{a}=\frac{2\ast N}{T-B}$. The perspective projection from the simulated camera’s viewport is given by $\mathbf{C}=\mathbf{P}\ast \mathbf{V}\ast \mathbf{W}$; where, $\mathbf{C}={\left[x_c{y}_c{z}_c{w}_c\right]}^T$ represents the image space coordinates and $\mathbf{W}={\left[x_w{y}_w{z}_w{w}_w\right]}^T$ represents the world coordinates. Finally, this camera projection is converted into normalized device coordinates (NDC) by performing a perspective divide (i.e., dividing throughout by $w_c$), obtaining a viewport projection by scaling and shifting the result, and then using the rasterization process of the graphics API (e.g., DirectX for Windows, Metal for macOS and Vulkan for Linux). Additionally, a post-processing step simulates the lens and film effects of the camera, such as lens distortion, depth of field, exposure, ambient occlusion, contact shadows, bloom, motion blur, film grain, and chromatic aberration.

4.3. Actuator Simulation

The vehicle is actuated using two driving actuators and a steering actuator, the response delays and saturation limits of which are matched with their real-world counterparts by tuning their torque profiles and actuation limits, respectively.

The driving actuators drive the rear wheels by applying a torque: ${}^i{\tau }_{drive}={}^i{I}_w\ast {}^i{\dot{\omega }}_w$, where ${}^i{I}_w=\frac{1}{2}\ast {}^i{m}_w\ast {}^i{r_w}^2$ is the moment of inertia, ${}^i{\dot{\omega }}_w$ is the angular acceleration, ${}^i{m}_w$ is the mass and ${}^i{r}_w$ is the radius of i-th wheel. Additionally, the holding torque of the driving actuators is simulated by applying an idle motor torque equivalent to the braking torque: ${}^i{\tau }_{idle}={}^i{\tau }_{brake}$.

The front wheels are steered using a steering actuator, which produces a torque proportional to the required angular acceleration: ${\tau }_{steer}=I_{steer}\ast {\dot{\omega }}_{steer}$. The individual turning angles, ${\delta }_l$ and ${\delta }_r$, for left and right wheels, respectively, are calculated based on the commanded steering angle $\delta$, using the Ackermann steering geometry defined by wheelbase l and track width w, as follows: $\left\{\begin{array}{c}{\delta }_l=ta{n}^{-1}\left(\frac{2\ast l\ast tan\left(\delta \right)}{2\ast l+w\ast tan\left(\delta \right)}\right)\\ {\delta }_r=ta{n}^{-1}\left(\frac{2\ast l\ast tan\left(\delta \right)}{2\ast l-w\ast tan\left(\delta \right)}\right)\end{array}\right.$.

4.4. Infrastructure Simulation

Simulated environments can be set up in one of the following ways:

• AutoDRIVE IDK: The modular and reconfigurable infrastructure development kit (IDK) can be used to create custom scenarios and maps by setting up the terrain modules, road networks, obstruction modules and traffic elements. These assets are present within the simulator source files. Particularly, the preconfigured maps depicted in Figure 3F(i,iii,iv) were constructed using the AutoDRIVE IDK.
• Plug-In Scenarios: AutoDRIVE Simulator supports third-party tools (e.g., RoadRunner [25]) and modular open-source architecture (MOSA) standards (e.g., OpenSCENARIO [26], OpenDRIVE [27], etc.) that enable extensibility. Additionally, users can import a wide array of plugins, packages and assets in a variety of industry-standard formats (FBX, OBJ, SKP, 3DS, USD, etc.) for developing or customizing driving scenarios. Furthermore, the graphics textures designed for AutoDRIVE Testbed can first be imported into the simulator before large-scale printing and real-world setup. The preconfigured map depicted in Figure 3F(ii) was designed using a third-party graphics’ editing software, imported in AutoDRIVE Simulator, and finally printed and set up using AutoDRIVE Testbed.
• Unity Terrain: Being built atop the Unity game engine, AutoDRIVE Simulator natively supports scenario design and development using Unity Terrain [28]. Users can define the terrain mesh, texture, heightmap, vegetation, skybox, wind, etc., to design on-road/off-road scenarios and perform variability testing.

At every time step, the simulator performs mesh–mesh interference detection and computes the contact forces, frictional forces and momentum transfer, along with the linear and angular drag acting on the rigid-bodies (refer Figure 4B).

4.5. Simulator Features

AutoDRIVE Simulator is developed atop the Unity [29] game engine, which employs PhysX [30] to simulate the multi-threaded framerate-independent kinematics and dynamics of all the physical entities and exploits the High-Definition Render Pipeline (HDRP) [31] along with the Post-Processing Stack [32] to render photorealistic graphics.

The simulator features an interactive graphical user interface (GUI) consisting of the Menu Panel on the left-hand side and Heads-Up Display (HUD) on the right-hand side. Figure 4C depicts the simulator’s GUI with both the panels enabled. The Menu Panel hosts input fields and buttons to configure and control various features of the simulator (refer Figure 4D). This includes controls for the communication bridge, along with a series of buttons for (a) toggling between manual and autonomous driving modes for the ego vehicle; (b) switching between the available scene cameras, with each providing a distinct view; (c) altering the graphics quality to match the quality–performance trade-off; (d) toggling the scene light to simulate day and night driving conditions; (e) resetting the scene to initial conditions; and (f) quitting the simulator application. The HUD Panel, on the other hand, displays prominent simulation parameters along with vehicle status and sensory data in real-time. It also hosts a time-synchronized data-recording functionality, which can be used to export vehicle as well as infrastructure data for a specific run, thereby fostering data-driven approaches aimed at autonomous driving and smart city management.

The simulator natively supports C# scripting, which can be leveraged to customize existing and/or introduce new features, functionalities, modules, behaviors, physics, graphics, communication bridges, and APIs, and even set up co-simulation frameworks with other simulation tools.

Finally, it is worth mentioning that the simulator, in its source form, was integrated with various plugins and packages, such as the Unity ML Agents Toolkit [33], a machine learning framework for developing and deploying deep imitation/reinforcement learning-based applications directly from within the simulator.

5. AutoDRIVE Devkit

AutoDRIVE Devkit is a collection of software packages, application programming interfaces (APIs) and tools, which enables the flexible development of autonomous driving, as well as smart city management algorithms targeting the testbed and/or simulator. It supports both local as well as distributed computing, thereby allowing for the development of both centralized and decentralized autonomy algorithms.

5.1. Autonomous Driving Software Stack

The autonomous driving software stack (ADSS) aids in the development of autonomy algorithms specifically targeting the vehicle. It can be used to develop single as well as multi-agent autonomous driving algorithms.

5.1.1. ROS Package

AutoDRIVE ROS package supports the flexible development of modular autonomy algorithms. It can be installed on an ROS-compatible workstation for interfacing with the simulator (locally/remotely), or directly on Nigel’s on-board computer for hardware deployment.

5.1.2. Scripting APIs

AutoDRIVE Devkit currently offers scripting APIs for Python and C++, which can be exploited to develop high-performance autonomy algorithms, without ROS as an intermediary. Such source codes can be interfaced with the simulator (locally/remotely), or directly deployed on Nigel’s on-board computer for hardware validation.

5.2. Smart City Software Stack

The smart city software stack (SCSS) aids in the development of autonomy algorithms specifically targeting the infrastructure. It can work in tandem with ADSS to develop smart city applications pertaining to traffic management.

5.2.1. SCM Server

AutoDRIVE Devkit offers a centralized Smart City Manager (SCM) server to monitor and control various “smart” elements. The server hosts a database to keep track of all the vehicles along with the active and passive traffic elements within a particular scene.

5.2.2. SCM Webapp

AutoDRIVE SCM hosts an interactive webapp, which allows for the users to connect with the database for monitoring and controlling the traffic flow in real-time.

6. Demonstration Case-Studies

This work showcases key features and capabilities of the AutoDRIVE Ecosystem through four carefully shortlisted case-studies (refer

Table 2. Decision matrix for choosing the demonstration case-studies.

Autonomy Algorithm	Platform Exploited	Development Framework	Autonomy Stack	Science and Technology Demonstrated	Agents Involved	Sensors Employed	Actuators Controlled
Autonomous Parking	AutoDRIVE Testbed	AutoDRIVE ROS Package (Python, C++)	Modular (Perception, Planning and Control)	Teleoperation, SLAM, Probabilistic Map-Based Localization, Global Planning, Local Planning, Motion Control, Static/Dynamic Collision Avoidance	Single-Agent System	LIDAR	Driving Actuators, Steering Actuator
Behavioral Cloning	AutoDRIVE Simulator, AutoDRIVE Testbed	AutoDRIVE Python API (Python)	End-to-End (Sensorimotor Policy)	Computer Vision, Deep Imitation Learning, Lane Keeping, Sim2Real Transition	Single-Agent System	Front Camera	Driving Actuators, Steering Actuator
Intersection Traversal	AutoDRIVE Simulator	Unity ML-Agents (C#)	End-to-End (Sensorimotor Policy)	V2V Communication, Deep Reinforcement Learning, Dynamic Collision Avoidance, Multi-Agent Cooperation and Coordination	Multi-Agent System	Incremental Encoders, IPS, IMU	Steering Actuator (Constant Throttle)
Smart City Management	AutoDRIVE Simulator	AutoDRIVE Webapp API (Python)	Modular (Surveillance, Planning and Control)	V2I Communication, IoT, Centralized Control and Coordination	Single-Agent System	None	Driving Actuators, Steering Actuator

). Although this paper cannot provide exhaustive details pertaining to any particular demonstration, we recommend that interested readers peruse this technical report [34].

It is to be noted that the presented demonstrations are by no means exhaustive and that AutoDRIVE Ecosystem can be employed to develop, simulate and deploy a much wider array of applications including (but not limited to) synthetic/real/hybrid data collection and labeling; traditional (deterministic/probabilistic, classical/optimal, etc.) as well as modern (deep imitation/reinforcement/hybrid learning, etc.) algorithms for perception, state estimation, path/motion-planning and motion control; modular as well as end-to-end autonomy stacks; benchmarking existing solutions (i.e., education) or innovating novel scientific and technological approaches for autonomy (i.e., research), etc.

6.1. Autonomous Parking

This demonstration leveraged AutoDRIVE’s ROS-enabled capabilities to demonstrate autonomous parking (refer Figure 5A). First, the vehicle mapped its surroundings using the Hector SLAM algorithm [35] (refer Figure 5B). It could then localize itself against this known static map using range-flow-based odometry [36] (refer Figure 5C) and an adaptive particle filter algorithm [37] (refer Figure 5D). For autonomous navigation, the vehicle planned a feasible global path from its current pose to parking pose using the A* algorithm [38], while also re-planning its local trajectory for dynamic collision avoidance using the timed-elastic-band approach [39]. A proportional controller generated driving (throttle/brake) and steering commands for the vehicle to track the local trajectory (refer Figure 5E).

Future work in this direction can include the benchmarking of various SOTA and/or novel algorithms for mapping, localization, path-planning and motion control.

6.2. Behavioral Cloning

This demonstration was based on [40], wherein the objective was to employ a convolutional neural network (CNN) [41] for cloning the end-to-end driving behavior of a human. As such, AutoDRIVE Simulator was exploited to record five laps’ worth of temporally coherent labeled manual driving data, which were balanced, augmented and pre-processed using standard computer vision techniques, to train a six-layer-deep CNN (refer Figure 6A). Post training for four epochs, with a learning rate of 1e-3 using the Adam optimizer [42], the training and validation losses converged stably without over/under-fitting. The trained model was then deployed back into the virtual world to validate its performance through activation and prediction analyses (refer Figure 6B).

Further, the same model was transferred to AutoDRIVE Testbed to validate the sim2real capability of the ecosystem through activation and prediction analyses (refer Figure 6C). To obtain a zero-shot sim2real transition: (i) the physical and visual aspects of virtual world were set up to be as close to the real world as possible, and (ii) the exhaustive data augmentation pipeline implicitly performed domain randomization. However, further investigation is required to comment on and improve the robustness of sim2real transfer considering sensor simulation, vehicle modeling and scenario representation. Finally, although this work adopted a coupled-control law for vehicle motion smoothing, a potential improvement could be to investigate independent actuation smoothing techniques such as low-pass filters or proximal bounds.

6.3. Intersection Traversal

Inspired by [43], this work demonstrates single and multi-agent (refer Figure 7A) intersection traversal using deep reinforcement learning (refer Figure 7B). Each agent collected a vectorized observation $o_t^i={\left[g^i,{\tilde{p}}^i,{\tilde{\psi }}^i,{\tilde{v}}^i\right]}_t$, including its relative goal location $g_t^i$, along with relative location ${\tilde{p}}_t^i$, relative yaw ${\tilde{\psi }}_t^i$ and velocity ${\tilde{v}}_t^i$ of its peers obtained through V2V communication. The action space of each agent, $a_t^i$, included discretized steering ${\delta }_t^i\in \left\{-1,0,1\right\}$ and constant throttle (${\tau }_t^i=80\%$). An extrinsic reward function $r_t^i=\left\{\begin{array}{c}{r}_{goal}=+1\hfill \\ r_{collision}=-0.425\ast {∥g_t^i∥}_2\hfill \end{array}\right.$ kept agents in check while training a three-layer, fully connected neural-network-based policy, ${\pi }_{\theta }\left(a_t|o_t\right)$ using a PPO algorithm [44] (refer Figure 7C). This ultimately resulted in all agents being able to safely traverse the intersection (refer Figure 7D). Although we could not implement this application in the real world owing to monetary constraints, investigating the sim2real transition of this application would be a natural progression of this work. Additionally, the application of actuation smoothing techniques could be a potential improvement.

6.4. Smart City Management

This novel use-case of smart city traffic management was possible due to AutoDRIVE’s V2I and IoT abilities. As depicted in Figure 8A), the SCM server hosted a database to keep track of all the traffic elements, and acted as a high-level behavior planner for the ego vehicle. It switched the vehicle behavior upon detecting respective traffic signs and lights by setting the appropriate throttle and steering trims, which were then passed on to a proximally optimal predictive (POP) controller coupled with adaptive longitudinal controller (ALC) [45]. Finally, the SCM server teleoperated the vehicle to achieve the mission objective (refer Figure 8B).

A natural progression of this work would be to implement a multi-agent scenario, preferably within a mixed-reality digital-twin setting, to investigate different strategies for efficient smart city traffic management.

7. Summary

AutoDRIVE was developed with the aim of tightly integrating real and virtual worlds into a common toolchain, without compromising the comprehensiveness, flexibility and accessibility required for prototyping and validating autonomy solutions. It has numerous applications, which are bound to increase as the ecosystem is upgraded. Potential improvements include supporting heterogeneous vehicles and robotic pedestrians, full-scale vehicles and environments, expanding API support and adding extended reality capabilities, to name a few. We hope that the community benefits from adopting this ecosystem for education, research or anything inbetween.