# Sensor-Model-Based Trajectory Optimization for UAVs to Enhance Detection Performance: An Optimal Control Approach and Experimental Results

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

#### 1.1. State of the Art

#### 1.2. Research Gap

#### 1.3. Research Problem

#### 1.4. Innovative Contribution and Novelty in This Work

#### 1.5. Outline

## 2. Materials and Methods

#### 2.1. Coverage Path Planning for Sensor Control

#### 2.2. Sensor Performance Models

#### 2.2.1. Deep Learning Based Object Detector

- The free code base of the YOLOv3 detector and the availability of public and annotated datasets (e.g., the UAVDT dataset [39]) have contributed to the widespread use of this detector.
- YOLOv3 enables real-time image-based object detection on commercially available hardware [40], which is especially advantageous for use on board the UAV.

**Figure 4.**Procedure for generating the sensor performance model utilizing a YOLOv3 object detector. States, the dataset, and the performance model are shown in light gray, actions are colored in blue. The green rectangle marks the environmental states, consisting of the atmospheric and photographic states.

#### 2.2.2. Machine Learning Based Object Classifier

- Atmospheric states: cloudiness, fog, precipitation and lightening conditions defined by the time of day and month.
- Topographic states: land cover (roads, meadow, water, vegetation and buildings) and the surface roughness within the sensor footprint.
- Photographic states: ground sample distance and the sensor elevation angle (see Figure 6 right plot).

#### 2.2.3. Perception Maps

#### 2.3. Optimal Control for UAV Trajectory Optimization

- The earth is assumed stationary and flat.
- The earth-fixed coordinate system is considered as an inertial system.
- The influence of wind or turbulence on the motion of the aircraft is neglected.
- The airspeed is predefined and can be considered approximately constant.
- The UAV is assumed to operate at a constant altitude, making the equation of motion for vertical motion obsolete.

#### 2.4. Trajectory Optimization with Nonlinear Model Predictive Control

- The solution of the OCP is obtained by closed-loop control. This allows for the compensation of uncertainties between the modeled system dynamics and the real system.
- Model predictive control is one of the few methods to handle hard system state and/or control input constraints [47].
- The course of the setpoints does not need to reproduce the system dynamics exactly.

- A suitable model must be found and modeled in order to be able to reproduce the system dynamics with sufficient accuracy.
- From the nonlinear system dynamics follows a general non-convex optimal control problem, for which only local optimal results can be computed [48].

#### 2.4.1. Fan-Shaped Path Planning

#### 2.4.2. Nonlinear Model Predictive Control

- The current system state ${x}_{n}$ at time ${t}_{n}$ is measured.
- The optimal control problem (22) is solved for the quadratic objective function (23) and the setpoint values ${x}_{n+k}^{ref}$ and ${u}_{n+k}^{ref}$. The result is the optimal control strategy ${\pi}^{\ast}\left({x}_{n}\right)$ with respect to the current state ${x}_{n}$.
- From the optimal control strategy ${\pi}^{\ast}\left({x}_{n}\right)$, the initial control input ${u}_{n}^{\ast}={\mu}_{n}^{\ast}\left({x}_{n}\right)$ is applied to the dynamical system for the duration of one time step $\Delta t$.
- At the end of the time step, the updated system state ${x}_{n+1}$ is measured at time ${t}_{n+1}$.
- The NMPC algorithm starts again at point 1 with the updated system state and continues until all time steps ${t}_{n}$ have been processed.

#### 2.4.3. Combining Path Planning and NMPC for Trajectory Optimization

#### 2.5. Trajectory Optimization with Dynamic Programming

#### 2.5.1. Dubins Path

- The velocity v of the UAV must be set constant.
- The maximum permissible roll angle ${\varphi}_{max}$ has to be defined.

#### 2.5.2. Dynamic Programming and Optimal Control

#### 2.5.3. Dubins Path Segments Smoothing

#### 2.6. Benchmark Trajectories

- Circle pattern.
- Racetrack pattern.
- Figure-8 pattern.

#### 2.7. Implementation

## 3. Results

#### 3.1. Validation Process and General Specifications

#### 3.2. Route Reconnaissance Scenario

#### 3.2.1. NMPC Trajectory Optimization

#### 3.2.2. DP&OC Trajectory Optimization

#### 3.3. Area Reconnaissance Scenario

#### 3.3.1. NMPC Trajectory Optimization

#### 3.3.2. DP&OC Trajectory Optimization

#### 3.4. Computational Effort

## 4. Discussion

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

AP | Average precision |

BLOB | Binary large object |

CPP | Coverage path planning |

CC | Classification cascade |

CC-SPM | Sensor performance model based on classification cascade object classifier |

DPM | Deformable part model |

DP&OC | Dynamic programming and optimal control |

FN | False negative |

FP | False positive |

GIS | Geographic information service |

GSD | Ground sample distance |

IoU | Intersection over union |

NMPC | Nonlinear model predictive control |

OCP | Optimal control problem |

PM | Perception map |

TM | Template matching |

TP | True positive |

UAV | Unmanned aerial vehicle |

Yolo-SPM | Sensor performance model based on YOLOv3 object detector |

## References

- Zhang, J.; Huang, H. Occlusion-Aware UAV Path Planning for Reconnaissance and Surveillance. Drones
**2021**, 5, 98. [Google Scholar] [CrossRef] - Cambone, S.A.; Krieg, K.; Pace, P.; Linton, W. Unmanned aircraft systems roadmap 2005–2030. Off. Secr. Def.
**2005**, 8, 4–15. [Google Scholar] - Avola, D.; Foresti, G.L.; Martinel, N.; Micheloni, C.; Pannone, D.; Piciarelli, C. Aerial video surveillance system for small-scale UAV environment monitoring. In Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy, 29 August–1 September 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1–6. [Google Scholar] [CrossRef]
- Manfreda, S.; McCabe, M.; Miller, P.; Lucas, R.; Pajuelo Madrigal, V.; Mallinis, G.; Ben Dor, E.; Helman, D.; Estes, L.; Ciraolo, G.; et al. On the Use of Unmanned Aerial Systems for Environmental Monitoring. Remote Sens.
**2018**, 10, 641. [Google Scholar] [CrossRef][Green Version] - Langhammer, J.; Vacková, T. Detection and Mapping of the Geomorphic Effects of Flooding Using UAV Photogrammetry. Pure Appl. Geophys.
**2018**, 175, 3223–3245. [Google Scholar] [CrossRef] - Nex, F.; Remondino, F. UAV for 3D mapping applications: A review. Appl. Geomat.
**2014**, 6, 1–15. [Google Scholar] [CrossRef] - Feraru, V.A.; Andersen, R.E.; Boukas, E. Towards an Autonomous UAV-based System to Assist Search and Rescue Operations in Man Overboard Incidents. In Proceedings of the 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Abu Dhabi, United Arab Emirates, 4–6 November 2020; pp. 57–64. [Google Scholar] [CrossRef]
- Qingqing, L.; Taipalmaa, J.; Queralta, J.P.; Gia, T.N.; Gabbouj, M.; Tenhunen, H.; Raitoharju, J.; Westerlund, T. Towards Active Vision with UAVs in Marine Search and Rescue: Analyzing Human Detection at Variable Altitudes. In Proceedings of the 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Abu Dhabi, United Arab Emirates, 4–6 November 2020; pp. 65–70. [Google Scholar] [CrossRef]
- Erdos, D.; Erdos, A.; Watkins, S.E. An experimental UAV system for search and rescue challenge. IEEE Aerosp. Electron. Syst. Mag.
**2013**, 28, 32–37. [Google Scholar] [CrossRef] - Sambolek, S.; Ivasic-Kos, M. Automatic Person Detection in Search and Rescue Operations Using Deep CNN Detectors. IEEE Access
**2021**, 9, 37905–37922. [Google Scholar] [CrossRef] - Jung, H.K.; Choi, G.S. Improved YOLOv5: Efficient Object Detection Using Drone Images under Various Conditions. Appl. Sci.
**2022**, 12, 7255. [Google Scholar] [CrossRef] - Howard, R.; Barrett, S.; Kunze, L. Don’t Blindly Trust Your CNN: Towards Competency-Aware Object Detection by Evaluating Novelty in Open-Ended Environments. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation, Xi’an, China, 30 May–5 June 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 13286–13292. [Google Scholar] [CrossRef]
- Zwick, M.; Gerdts, M.; Stütz, P. Sensor Model-Based Trajectory Optimization for UAVs Using Nonlinear Model Predictive Control. In Proceedings of the AIAA SCITECH 2022 Forum; American Institute of Aeronautics and Astronautics, San Diego, CA, USA, 3–7 January 2022. [Google Scholar] [CrossRef]
- Zwick, M.; Gerdts, M.; Stutz, P. Enhancing Detection Performance through Sensor Model-based Trajectory Optimization for UAVs. In Proceedings of the 2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC), San Antonio, TX, USA, 3–7 October 2021; pp. 1–10. [Google Scholar] [CrossRef]
- Ru, P.; Subbarao, K. Nonlinear Model Predictive Control for Unmanned Aerial Vehicles. Aerospace
**2017**, 4, 31. [Google Scholar] [CrossRef] - Kamel, M.; Stastny, T.; Alexis, K.; Siegwart, R. Model Predictive Control for Trajectory Tracking of Unmanned Aerial Vehicles Using Robot Operating System. In Robot Operating System (ROS); Koubaa, A., Ed.; Studies in Computational Intelligence; Springer International Publishing: Cham, Switzerland, 2017; Volume 707, pp. 3–39. [Google Scholar] [CrossRef]
- Garcia, G.A.; Keshmiri, S.S.; Stastny, T. Robust and Adaptive Nonlinear Model Predictive Controller for Unsteady and Highly Nonlinear Unmanned Aircraft. IEEE Trans. Control Syst. Technol.
**2015**, 23, 1620–1627. [Google Scholar] [CrossRef] - Zhang, Y.; Wang, W.; Huang, P.; Jiang, Z. Monocular Vision-based Sense and Avoid of UAV Using Nonlinear Model Predictive Control. Robotica
**2019**, 37, 1582–1594. [Google Scholar] [CrossRef] - Ahmed, K.; Bousson, K.; Coelho, M.d.F. A Modified Dynamic Programming Approach for 4D Minimum Fuel and Emissions Trajectory Optimization. Aerospace
**2021**, 8, 135. [Google Scholar] [CrossRef] - Quintero, S.A.P.; Papi, F.; Klein, D.J.; Chisci, L.; Hespanha, J.P. Optimal UAV coordination for target tracking using dynamic programming. In Proceedings of the 49th IEEE Conference on Decision and Control (CDC), Atlanta, GA, USA, 15–17 December 2010; pp. 4541–4546. [Google Scholar] [CrossRef][Green Version]
- Harada, A.; Miyazawa, Y. Dynamic Programming Applications to Flight Trajectory Optimization. IFAC Proc. Vol.
**2013**, 46, 441–446. [Google Scholar] [CrossRef] - MOKRANE, A.; BRAHAM, A.C.; CHERKI, B. UAV Path Planning Based on Dynamic Programming Algorithm On Photogrammetric DEMs. In Proceedings of the 2020 International Conference on Electrical Engineering (ICEE), Istanbul, Turkey, 25–27 September 2020; Recioui, A., Ed.; IEEE: Piscataway, NJ, USA, 2020; pp. 1–5. [Google Scholar] [CrossRef]
- Goerzen, C.; Kong, Z.; Mettler, B. A Survey of Motion Planning Algorithms from the Perspective of Autonomous UAV Guidance. J. Intell. Robot. Syst.
**2010**, 57, 65–100. [Google Scholar] [CrossRef] - Betts, J.T. Survey of Numerical Methods for Trajectory Optimization. J. Guid. Control Dyn.
**1998**, 21, 193–207. [Google Scholar] [CrossRef][Green Version] - Hellert, C.; Koch, S.; Stutz, P. Using Algorithm Selection for Adaptive Vehicle Perception Aboard UAV. In Proceedings of the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan, 18–21 September 2019; pp. 1–8. [Google Scholar] [CrossRef]
- Acatay, O.; Sommer, L.; Schumann, A.; Beyerer, J. Comprehensive Evaluation of Deep Learning based Detection Methods for Vehicle Detection in Aerial Imagery. In Proceedings of the 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand, 27–30 November 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Andriluka, M.; Schnitzspan, P.; Meyer, J.; Kohlbrecher, S.; Petersen, K.; von Stryk, O.; Roth, S.; Schiele, B. Vision based victim detection from unmanned aerial vehicles. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, 18–22 October 2010; IEEE: Piscataway, NJ, USA, 2010; pp. 1740–1747. [Google Scholar] [CrossRef][Green Version]
- Liu, Y.; Han, K.; Rasdorf, W. Assessment and Prediction of Impact of Flight Configuration Factors on UAS-Based Photogrammetric Survey Accuracy. Remote Sens.
**2022**, 14, 4119. [Google Scholar] [CrossRef] - Russ, M.; Stütz, P. Airborne sensor and perception management: A conceptual approach for surveillance UAS. In Proceedings of the 2012 15th International Conference on Information Fusion, Singapore, 9–12 July 2012; pp. 2444–2451. [Google Scholar]
- Sandino, J.; Vanegas, F.; Gonzalez, F.; Maire, F. Autonomous UAV Navigation for Active Perception of Targets in Uncertain and Cluttered Environments. In Proceedings of the 2020 IEEE Aerospace Conference, Big Sky, MT, USA, 7–14 March 2020; IEEE: Piscataway, NJ, USA, 2020. [Google Scholar] [CrossRef]
- Stecz, W.; Gromada, K. Determining UAV Flight Trajectory for Target Recognition Using EO/IR and SAR. Sensors
**2020**, 20, 5712. [Google Scholar] [CrossRef] - Erickson, L.; LaValle, S. A Simple, but NP-Hard, Motion Planning Problem. In Proceedings of the AAAI Conference on Artificial Intelligence, Bellevue, WA, USA, 14–18 July 2013; Volume 27, pp. 1388–1393. [Google Scholar] [CrossRef]
- Chandler, P.R.; Pachter, M. Research issues in autonomous control of tactical UAVs. In Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No.98CH36207), Philadelphia, PA, USA, 26 June 1998; Volume 1, pp. 394–398. [Google Scholar] [CrossRef][Green Version]
- Choset, H.; Pignon, P. Coverage Path Planning: The Boustrophedon Cellular Decomposition. In Field and Service Robotics; Zelinsky, A., Ed.; Springer: London, UK, 1998; pp. 203–209. [Google Scholar] [CrossRef][Green Version]
- Choset, H. Coverage for robotics—A survey of recent results. Ann. Math. Artif. Intell.
**2001**, 31, 113–126. [Google Scholar] [CrossRef] - Krump, M.; Stütz, P. UAV Based Vehicle Detection with Synthetic Training: Identification of Performance Factors Using Image Descriptors and Machine Learning. In Modelling and Simulation for Autonomous Systems; Mazal, J., Fagiolini, A., Vasik, P., Turi, M., Eds.; Springer International Publishing: Cham, Switzerland, 2021; pp. 62–85. [Google Scholar] [CrossRef]
- Jiao, L.; Zhang, F.; Liu, F.; Yang, S.; Li, L.; Feng, Z.; Qu, R. A Survey of Deep Learning-Based Object Detection. IEEE Access
**2019**, 7, 128837–128868. [Google Scholar] [CrossRef] - Zhao, Z.Q.; Zheng, P.; Xu, S.T.; Wu, X. Object Detection With Deep Learning: A Review. IEEE Trans. Neural Netw. Learn. Syst.
**2019**, 30, 3212–3232. [Google Scholar] [CrossRef][Green Version] - Du, D.; Qi, Y.; Yu, H.; Yang, Y.; Duan, K.; Li, G.; Zhang, W.; Huang, Q.; Tian, Q. The unmanned aerial vehicle benchmark: Object detection and tracking. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 370–386. [Google Scholar]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv
**2018**, arXiv:1804.02767. [Google Scholar] - Krump, M.; Ruß, M.; Stütz, P. Deep Learning Algorithms for Vehicle Detection on UAV Platforms: First Investigations on the Effects of Synthetic Training. In Modelling and Simulation for Autonomous Systems; Mazal, J., Fagiolini, A., Vasik, P., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 50–70. [Google Scholar] [CrossRef]
- Zwick, M.; Koch, S.; Stütz, P. Enhancing Detection and Tracking Performance Using Sensor-specific Flight Trajectory Generation for UAVs: A Conceptual Approach. In Proceedings of the AIAA Scitech 2020 Forum, American Institute of Aeronautics and Astronautics. Orlando, FL, USA, 6–10 January 2020. [Google Scholar] [CrossRef]
- Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv
**2020**. [Google Scholar] [CrossRef] - Nepal, U.; Eslamiat, H. Comparing YOLOv3, YOLOv4 and YOLOv5 for Autonomous Landing Spot Detection in Faulty UAVs. Sensors
**2022**, 22, 464. [Google Scholar] [CrossRef] [PubMed] - Hellert, C. Algorithmenauswahl für den Adaptiven Sensoreinsatz an Bord Unbemannter Luftfahrzeuge. Ph.D. Thesis, Universität der Bundeswehr München, Neubiberg, Germany, 2019. [Google Scholar]
- Liu, Y.; Zhao, Y. A virtual-waypoint based artificial potential field method for UAV path planning. In Proceedings of the 2016 IEEE Chinese Guidance, Navigation and Control Conference (CGNCC), Nanjing, China, 12–14 August 2016; pp. 949–953. [Google Scholar] [CrossRef]
- Allgöwer, F.; Badgwell, T.A.; Qin, J.S.; Rawlings, J.B.; Wright, S.J. Nonlinear Predictive Control and Moving Horizon Estimation—An Introductory Overview. In Advances in Control; Frank, P.M., Ed.; Springer: London, UK, 1999; pp. 391–449. [Google Scholar] [CrossRef]
- Mayne, D. Nonlinear Model Predictive Control: Challenges and Opportunities. In Nonlinear Model Predictive Control; Birkhäuser: Basel, Switzerland, 2000; pp. 23–44. [Google Scholar] [CrossRef]
- Grüne, L.; Pannek, J. Nonlinear Model Predictive Control; Springer: Cham, Switzerland, 2017. [Google Scholar] [CrossRef]
- Kraft, D. On Converting Optimal Control Problems. In Computational Mathematical Programming; Springer: Berlin/Heidelberg, Germany, 1985; pp. 261–280. [Google Scholar] [CrossRef]
- Bock, H.G.; Plitt, K.J. A Multiple Shooting Algorithm for Direct Solution of Optimal Control Problems. IFAC Proc. Vol.
**1984**, 17, 1603–1608. [Google Scholar] [CrossRef] - Gerdts, M. Direct Shooting Method for the Numerical Solution of Higher-Index DAE Optimal Control Problems. J. Optim. Theory Appl.
**2003**, 117, 267–294. [Google Scholar] [CrossRef] - Schittkowski, K.; Yuan, Y.X. Sequential Quadratic Programming Methods. In Wiley Encyclopedia of Operations Research and Management Science; Cochran, J.J., Ed.; Wiley Interscience: Hoboken, NJ, USA, 2011. [Google Scholar] [CrossRef]
- de Nicolao, G.; Magni, L.; Scattolini, R. Stability and Robustness of Nonlinear Receding Horizon Control. In Nonlinear Model Predictive Control; Birkhäuser: Basel, Switzerland, 2000; pp. 3–22. [Google Scholar] [CrossRef]
- Mayne, D.Q.; Rawlings, J.B.; Rao, C.V.; Scokaert, P. Constrained model predictive control: Stability and optimality. Automatica
**2000**, 36, 789–814. [Google Scholar] [CrossRef] - Allgöwer, F.; Findeisen, R.; Nagy, Z.K. Nonlinear model predictive control: From theory to application. J. Chin. Inst. Chem. Eng.
**2004**, 35, 299–315. [Google Scholar] - Findeisen, R.; Allgöwer, F. An introduction to nonlinear model predictive control. In Proceedings of the 21st Benelux Meeting on Systems and Control, 2002, Veldhoven, The Netherlands, 19–21 March 2002; Volume 11, pp. 119–141. [Google Scholar]
- Dubins, L.E. On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents. Am. J. Math.
**1957**, 79, 497. [Google Scholar] [CrossRef] - Reeds, J.; Shepp, L. Optimal paths for a car that goes both forwards and backwards. Pac. J. Math.
**1990**, 145, 367–393. [Google Scholar] [CrossRef][Green Version] - Chitsaz, H.; LaValle, S.M. Time-optimal paths for a Dubins airplane. In Proceedings of the 2007 46th IEEE Conference on Decision and Control, New Orleans, LA, USA, 12–14 December 2007; pp. 2379–2384. [Google Scholar] [CrossRef][Green Version]
- Lugo-Cardenas, I.; Flores, G.; Salazar, S.; Lozano, R. Dubins path generation for a fixed wing UAV. In Proceedings of the 2014 International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA, 27–30 May 2014; pp. 339–346. [Google Scholar] [CrossRef][Green Version]
- Owen, M.; Beard, R.W.; McLain, T.W. Implementing Dubins Airplane Paths on Fixed-Wing UAVs. In Handbook of Unmanned Aerial Vehicles; Valavanis, K.P., Vachtsevanos, G.J., Eds.; Springer: Dordrecht, The Netherlands, 2015; pp. 1677–1701. [Google Scholar] [CrossRef][Green Version]
- LaValle, S.M. Planning Algorithms; Cambridge University Press: Cambridge, UK, 2006. [Google Scholar] [CrossRef][Green Version]
- Shkel, A.M.; Lumelsky, V. Classification of the Dubins set. Robot. Auton. Syst.
**2001**, 34, 179–202. [Google Scholar] [CrossRef] - Boissonnat, J.D.; Cerezo, A.; Leblond, J. Shortest paths of bounded curvature in the plane. In Proceedings of the Proceedings 1992 IEEE International Conference on Robotics and Automation, Nice, France, 12–14 May 1992; pp. 2315–2320. [Google Scholar] [CrossRef]
- Bellman, R. Dynamic Programming, 6th ed.; University of Princeton: Princeton, NJ, USA, 1972. [Google Scholar]
- Bertsekas, D.P. Dynamic Programming and Optimal Control, 4th ed.; Athena scientific optimization and computation series; Athena Scientific: Belmont, MA, USA, 2017; Volume 1. [Google Scholar]
- Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.; Foote, T.; Leibs, J.; Berger, E.; Wheeler, R.; Ng, A.Y. ROS: An open-source Robot Operating System. In Proceedings of the ICRA Workshop on Open Source Software, 2009, Kobe, Japan, 12–17 May 2009; Volume 3, p. 5. [Google Scholar]

**Figure 1.**Illustration of relevant influencing factors on sensor-model-based trajectory optimization. Adapted from [13].

**Figure 2.**Principle of coverage path planning for a reconnaissance area (green). The sensor footprint path defines the positioning of the individual sensor footprints (pale blue). The size of the footprint is defined by ${w}_{fp}$ and the Euclidean distance between footprints is determined by ${d}_{fp}$. The black dotted line marks the scanned area.

**Figure 3.**The sensor performance model maps selected environmental states to the expected detection performance of a specific perception chain (not displayed). These environmental states comprise atmospheric, photographic, and topographic conditions resulting from the positioning of the UAV and the sensor footprint on the ground.

**Figure 5.**Illustration of the average precision (color-coded) for different interval ranges of the ground sample distance and the elevation angle corresponding to a specific composition of the environmental state.

**Figure 6.**Representation of a perception map from the CC sensor performance model as a 3-dimensional plot (

**left**) and the same map in a planar representation (

**right**), with reference to the corresponding sensor footprint (pale blue square) on the ground. The elevation angle $\alpha $ is determined by the horizontal distance ${d}_{hor}$ and the altitude above ground ${h}_{agl}$. The color-coding of the perception map corresponds to the predicted detection performance. Light colors represent high performance values, while darker colors correlate with lower values.

**Figure 8.**Depiction of the principle of path planning. The fan-shaped path array consists (for representational reasons) of 9 evenly spaced curves (thin black lines). The thick black line is the resulting UAV trajectory from trajectory optimization. The square represents the sensor footprint on the ground. The perception map, which results from atmospheric and topographic conditions is color-coded. Yellow areas mark regions with high detection performance. Adapted from [13].

**Figure 9.**Example of a Dubins path from the start configuration a to the goal configuration b defined by a specific set of motion primitives L, S and R (

**left**). The associated curvature profile of the Dubins path is plotted on the (

**right**).

**Figure 11.**Illustration of the states and the state transitions in an acyclic graph. Circles represent the states $0,\dots ,{n}_{k}$ in the individual time steps $0,\dots ,N$. The arrows represent the state transitions between two states. As an example, the state-dependent costs ${c}_{st,k}^{i}$ and the state transition costs ${c}_{tr,k}^{ij}$ are plotted from time step k to $k+1$.

**Figure 12.**Illustration of the benchmark trajectories Circle (

**left**), Racetrack (

**center**) and Figure-8 (

**right**). Additionally, the starting point ${p}_{start}$, the support point ${p}_{sup}$ and the path direction are sketched. The radius ${r}_{loiter}$ is predefined or results from the design.

**Figure 13.**Illustration of the route reconnaissance scenario. The green line marks the reconnaissance route, supplemented by several perception maps resulting from the CC-SPM performance model. The color-coding of the different perception maps corresponds to the predicted detection performance. Light colors represent high performance values, while darker colors correlate with lower values.

**Figure 15.**Illustration of the detection performances for the NMPC and benchmark trajectory with respect to the sensor footprint path length. The black line marks the theoretical maximum detection performance as an upper bound.

**Figure 16.**Trajectory optimization for the route reconnaissance scenario with sensor performance model CC-SPM in (

**a**,

**c**) and with Yolo-SPM in (

**b**,

**d**). The blue line indicates the NMPC-optimized reference trajectory and the light green line represents the benchmark trajectory. The starting points of both trajectories are identical and marked by a black aircraft symbol. In (

**a**,

**b**), the Racetrack benchmark pattern is displayed, whereas in (

**c**,

**d**), the Figure-8 pattern is applied.

**Figure 17.**Illustration of the control inputs “roll rate” and “acceleration” for route reconnaissance with CC-SPM, plotted with respect to the flight duration. The control inputs yield changes in the system states “velocity” and “roll angle”. Shown also are the predefined limitations.

**Figure 18.**Depiction of the DP&OC-optimized trajectories for the route reconnaissance scenario with sensor performance models CC-SPM (

**left**) and Yolo-SPM (

**right**). The blue line marks the UAV flight trajectory and the green line maps the sensor footprint on the ground.

**Figure 19.**Illustration of the area reconnaissance scenario. The green line marks the sensor footprint path within the green reconnaissance area. Several perception maps resulting from the CC-SPM model illustrate the detection performance along the footprint path. The red lines indicate the positions of the perception maps along the sensor path. The color-coding of the different perception maps corresponds to the predicted detection performance.

**Figure 21.**Trajectory optimization for the area reconnaissance scenario with sensor performance model CC-SPM in (

**a**,

**c**) and with Yolo-SPM in (

**b**,

**d**). The blue line indicates the NMPC-optimized reference trajectory and the light green line marks the benchmark trajectory. The starting points of both trajectories are identical and depicted by the aircraft symbol. In (

**a**,

**b**), the Figure-8 pattern is used, whereas in (

**c**,

**d**), the Circle benchmark pattern is applied.

**Figure 22.**Illustration of the DP&OC-optimized trajectories for the area reconnaissance scenario with sensor performance models CC-SPM (

**left**) and Yolo-SPM (

**right**). The blue line depicts the UAV flight trajectory and the green line marks the sensor footprint on the ground.

Parameter | Setting | Remark |
---|---|---|

target ground sample distance $gs{d}_{ref}$ | 0.07 m | predefined |

sensor resolution ${R}_{sens}$ | 1920 px | predefined |

sweep width ${w}_{fp}$ | 134.4 m | from Equation (1) |

footprint velocity ${v}_{fp}$ | 30 m/s | predefined |

time step interval $\Delta t$ | 0.5 s | predefined |

distance ${d}_{fp}$ | 15 m | from Equation (2) |

**Table 2.**Atmospheric states comprising the aerial imagery dataset of [36].

Environmental State | Attributes |
---|---|

season | summer, autumn |

daytime | day, night |

visibility | clear, foggy |

road condition | wet, dry |

sky cover | covered, sunny |

Parameter | Setting | Remark |
---|---|---|

UAV altitude above ground ${h}_{agl}$ | 500 m | predefined |

perception map diameter ${d}_{pm}$ | 2000 m | predefined |

Parameter | Setting | Remark |
---|---|---|

preview horizon time steps ${M}_{prev}$ | 25 | predefined |

time step interval $\Delta t$ | 0.5 s | from Table 1 |

uav setpoint velocity ${v}_{ref}$ | 35 m/s | predefined |

path length ${l}_{path}$ | 437.5 m | from Equation (13) |

number of paths Z | 15 | predefined |

Parameter | Setting | Remark |
---|---|---|

prediction horizon N | 10 | predefined |

maximum roll angle ${\varphi}_{max}$ | 0.7 rad | from Equation (18) |

setpoint roll angle ${\varphi}^{ref}$ | 0 rad | predefined |

maximum roll rate ${\omega}_{max}$ | 0.5 rad/s | from Equation (19) |

setpoint roll rate ${\omega}^{ref}$ | 0 rad/s | predefined |

minimum velocity ${v}_{min}$ | 33 m/s | from Equation (21) |

maximum velocity ${v}_{max}$ | 37 m/s | from Equation (21) |

setpoint velocity ${v}^{ref}$ | 35 m/s | predefined |

maximum acceleration ${a}_{max}$ | 0.1 $\mathrm{m}/{\mathrm{s}}^{2}$ | from Equation (20) |

setpoint acceleration ${a}^{ref}$ | 0 $\mathrm{m}/{\mathrm{s}}^{2}$ | predefined |

diagonal weighting matrix Q | 1, 1, 0.1, 0.1, 0.1 | predefined |

diagonal weighting matrix R | 0.5, 0.5 | predefined |

Parameter | Setting | Remark |
---|---|---|

weighting factor $\gamma $ | 0.8 | predefined |

Parameter | Setting | Remark |
---|---|---|

UAV velocity v | 35 m/s | predefined |

maximum roll angle ${\varphi}_{max}$ | 0.694 rad | predefined |

gravitational acceleration g | 9.81 $\mathrm{m}/{\mathrm{s}}^{2}$ | |

minimum turn radius ${r}_{min}$ | 150 m | from Equation (35) |

Parameter | Setting | Remark |
---|---|---|

UAV velocity v | 35 m/s | from Table 7 |

minimum turn radius ${r}_{min}$ | 150 m | from Table 7 |

time step interval $\Delta t$ | 0.5 s | from Table 1 |

minimum Dubins path length ${s}_{min}$ | 17.5 m | from Equation (44) |

maximum Dubins path length ${s}_{max}$ | 942.5 m | from Equation (45) |

weighting factor ${p}_{w}$ | 0.5 | predefined |

number of yaw angles ${m}_{\psi ,n}$ | 12 | predefined |

Parameter | Setting | Remark |
---|---|---|

time of day | 16 h | predefined |

month | June | predefined |

cloud cover | 25% | predefined |

fog density | 0% | predefined |

precipitation | 0% | predefined |

**Table 10.**Parameter settings of the atmospheric conditions for the Yolo-SPM sensor performance model.

Parameter | Setting | Remark |
---|---|---|

daytime | day | predefined |

season | autumn | predefined |

visibility | clear | predefined |

road condition | wet | predefined |

sky cover | covered | predefined |

CC-SPM | Yolo-SPM | |||
---|---|---|---|---|

NMPC | Benchm. | NMPC | Benchm. | |

maximum average detection performance | 0.972 | 0.972 | 0.936 | 0.936 |

average detection performance (abs.) | 0.815 | 0.772 | 0.924 | 0.878 |

average detection performance (rel.) | 83.88% | 79.42% | 98.66% | 93.77% |

CC-SPM | Yolo-SPM | |
---|---|---|

maximum average detection performance | 0.972 | 0.936 |

average detection performance (abs.) | 0.910 | 0.936 |

average detection performance (rel.) | 93.62% | 100.00% |

CC-SPM | Yolo-SPM | |||
---|---|---|---|---|

NMPC | Benchm. | NMPC | Benchm. | |

maximum average detection performance | 0.948 | 0.948 | 0.936 | 0.936 |

average detection performance (abs.) | 0.846 | 0.810 | 0.933 | 0.887 |

average detection performance (rel.) | 89.19% | 85.48% | 99.66% | 94.80% |

CC-SPM | Yolo-SPM | |
---|---|---|

maximum average detection performance | 0.948 | 0.936 |

average detection performance (abs.) | 0.909 | 0.936 |

average detection performance (rel.) | 95.89% | 100.00% |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zwick, M.; Gerdts, M.; Stütz, P. Sensor-Model-Based Trajectory Optimization for UAVs to Enhance Detection Performance: An Optimal Control Approach and Experimental Results. *Sensors* **2023**, *23*, 664.
https://doi.org/10.3390/s23020664

**AMA Style**

Zwick M, Gerdts M, Stütz P. Sensor-Model-Based Trajectory Optimization for UAVs to Enhance Detection Performance: An Optimal Control Approach and Experimental Results. *Sensors*. 2023; 23(2):664.
https://doi.org/10.3390/s23020664

**Chicago/Turabian Style**

Zwick, Markus, Matthias Gerdts, and Peter Stütz. 2023. "Sensor-Model-Based Trajectory Optimization for UAVs to Enhance Detection Performance: An Optimal Control Approach and Experimental Results" *Sensors* 23, no. 2: 664.
https://doi.org/10.3390/s23020664