# Fuzzy Gain-Scheduling Based Fault Tolerant Visual Servo Control of Quadrotors

## Abstract

**:**

## 1. Introduction

**s**according to their metrics such as coordinates in the image plane. The main goal is to reach the desired

**s**and the VS control law in terms of the velocities of the robotic system, especially the Cartesian velocities of the end effector, defined by the error vector between

^{*}**s**and

**s**. After this generalized definition of VS, the approaches can be classified into two main types: position-based visual servoing (PBVS) and image-based visual servoing (IBVS) [1]. On the one hand, in PBVS, the control law uses the information about the pose of the end effector relative to the desired pose to obtain

^{*}**s**. On the other hand, IBVS utilizes

^{*}**s**obtained from the instant image without any operations. PBVS is affected by pose estimation errors and IBVS is subject to keeping the image features in the field of view (FOV); therefore, hybrid approaches such as partitioned and 2½ D VS [6,7] or featureless approaches such as kernel-based VS [8] have been derived to avoid these problems. The advantage of robustness against depth estimation errors makes IBVS worthy for practical applications and it is preferred in this study.

## 2. The Proposed Fault Tolerant Visual Servo Control System

_{i}, v

_{i}are the coordinates in the u-v image plane. To characterize the behavior of the fixed-motionless feature points, these vectors are merged into a matrix in the form:

**s**having the same dimensions as the desired fixed feature points matrix. Error convergence is the primary objective of all VS approaches:

^{*}**Assumption**

**1.**

**Assumption**

**2.**

**I**is the inertia matrix,

**ω**is the angular velocity vector, and τ is the torque vector applied to the quadrotor. The motion of the quadrotor is defined by the relation between torque vector and height control force using the torque applied to each propeller:

**s**as the input of the bank of fault approximators. Each approximator approximates the LOE of a propeller as f

^{*}_{i}.

_{ij}is the weight from neuron i to neuron j, and η is the learning rate parameter. Instead of this parameter update, LM uses the sum of square errors for each input sample as the first step and its first derivative is defined in partial differential form as the Jacobian matrix in Equation (14). Then, the parameter update is implemented with μ learning rate using Equation (15):

_{n}is the output dataset, x

_{n}is the input dataset, and ε is the maximum error, as shown in Figure 4. The line segments for the boundary linear equations are shown in red and the defined linear equation by linear SVM is shown in blue. Using basic quadratic programming approaches, this minimization problem may be stated in ordinary quadratic programming form. However, using quadratic programming approaches might be computationally costly. Therefore, approaches such as sequential minimal optimization (SMO) are referred to [31]. For input–output mapping for fault detection, four independent SVMs, each for one fault output, should be defined.

**T**:

_{i}is the cell state that is the memory of LSTM, h

_{i}is the hidden state, and x

_{t}is the input. The network can more successfully learn long-term associations in the data thanks to the extra gates. LSTM networks are superior to simple RNNs for evaluating sequential data because they are less sensitive to the time gap.

## 3. Simulation Results

#### 3.1. Case 1: Fixed Target Features under Noise

#### 3.2. Case 2: Moving Target Features under Noise

## 4. Conclusions

## Supplementary Materials

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Conflicts of Interest

## References

- Chaumette, F.; Hutchinson, S. Visual Servo Control. I. Basic Approaches. IEEE Robot. Autom. Mag.
**2006**, 13, 82–90. [Google Scholar] [CrossRef] - Chaumette, F.; Hutchinson, S. Visual Servo Control. II. Advanced Approaches. IEEE Robot. Autom. Mag.
**2007**, 14, 109–118. [Google Scholar] [CrossRef] - Agravante, D.J.; Claudio, G.; Spindler, F.; Chaumette, F. Visual Servoing in an Optimization Framework for the Whole-Body Control of Humanoid Robots. IEEE Robot. Autom. Lett.
**2017**, 2, 608–615. [Google Scholar] [CrossRef] [Green Version] - Pebrianti, D.; Kendoul, F.; Azrad, S.; Wang, W.; Nonami, K. Autonomous Hovering and Landing of a Quad-Rotor Micro Aerial Vehicle by Means of on Ground Stereo Vision System. J. Syst. Des. Dyn.
**2010**, 4, 269–284. [Google Scholar] [CrossRef] [Green Version] - Corke, P.I. Robotics, Vision and Control; Fundamental Algorithms in MATLAB, 2nd ed.; Springer: Berlin/Heidelberg, Germany, 2017. [Google Scholar]
- Collewet, C.; Marchand, E.; Chaumette, F. Visual Servoing Set Free from Image Processing. In Proceedings of the 2008 IEEE International Conference on Robotics and Automation, Pasadena, CA, USA, 19–23 May 2008; pp. 81–86. [Google Scholar] [CrossRef] [Green Version]
- Malis, E.; Chaumette, F.; Boudet, S. 2 1/2 D Visual Servoing. IEEE Trans. Robot. Autom.
**1999**, 15, 238–250. [Google Scholar] [CrossRef] [Green Version] - Kallem, V.; Swensen, J.P.; Hager, G.D.; Cowan, N.J. Kernel-Based Visual Servoing. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, 29 October–2 November 2007; pp. 1975–1980. [Google Scholar]
- Mahony, R.; Hamel, T. Image-Based Visual Servo Control of Aerial Robotic Systems Using Linear Image Features. IEEE Trans. Robot.
**2005**, 21, 227–239. [Google Scholar] [CrossRef] - Ceren, Z.; Altug, E. Image Based and Hybrid Visual Servo Control of an Unmanned Aerial Vehicle. J. Intell. Robot. Syst.
**2012**, 65, 325–344. [Google Scholar] [CrossRef] - Odile, B.; Mahony, R.; Guenard, N.; Chaumette, F.; Hamel, T.; Eck, L. Kinematic Visual Servo Control of a Quadrotor Aerial Vehicle. IEEE Trans. Robot.
**2007**, 25, 833–838. [Google Scholar] - Hamel, T.; Mahony, R. Image Based Visual Servo-Control for a Class of Aerial Robotic Systems. Automatica
**2007**, 43, 1975–1983. [Google Scholar] [CrossRef] [Green Version] - Plinval, H.; Morin, P.; Mouyon, P.; Hamel, T. Visual Servoing for Underactuated VTOL UAVs: A Linear, Homography-Based Framework. Int. J. Robust Nonlinear Control
**2014**, 24, 2285–2308. [Google Scholar] [CrossRef] [Green Version] - Mebarki, R.; Lippiello, V.; Siciliano, B. Nonlinear Visual Control of Unmanned Aerial Vehicles in GPS-Denied Environments. IEEE Trans. Robot.
**2015**, 31, 1004–1017. [Google Scholar] [CrossRef] - Abdessameud, A.; Janabi-Sharifi, F. Image-Based Tracking Control of VTOL Unmanned Aerial Vehicles. Automatica
**2015**, 53, 111–119. [Google Scholar] [CrossRef] - Zheng, D.; Wang, H.; Wang, J.; Zhang, X.; Chen, W. Toward Visibility Guaranteed Visual Servoing Control of Quadrotor UAVs. IEEE/ASME Trans. Mechatron.
**2019**, 24, 1087–1095. [Google Scholar] [CrossRef] - Zhang, K.; Shi, Y.; Sheng, H. Robust Nonlinear Model Predictive Control Based Visual Servoing of Quadrotor UAVs. IEEE/ASME Trans. Mechatron.
**2021**, 26, 700–708. [Google Scholar] [CrossRef] - Liu, H.; Liu, D.; Lyu, Y. Completely Distributed Time-Varying Formation Target Tracking for Quadrotor Team via Image-Based Visual Servoing. IEEE Trans. Veh. Technol.
**2022**, 71, 21–32. [Google Scholar] [CrossRef] - Xie, H.; Lynch, A.F. Input Saturated Visual Servoing for Unmanned Aerial Vehicles. IEEE/ASME Trans. Mechatron.
**2017**, 22, 952–960. [Google Scholar] [CrossRef] - Zhao, W.; Liu, H.; Lewis, F.L.; Valavanis, K.P.; Wang, X. Robust Visual Servoing Control for Ground Target Tracking of Quadrotors. IEEE Trans. Control Syst. Technol.
**2020**, 28, 1980–1987. [Google Scholar] [CrossRef] - Cao, Z.; Chen, X.; Yu, Y.; Yu, J.; Liu, X.; Zhou, C.; Tan, M. Image Dynamics-Based Visual Servoing for Quadrotors Tracking a Target with a Nonlinear Trajectory Observer. IEEE Trans. Syst. Man Cybern. Syst.
**2020**, 50, 376–384. [Google Scholar] [CrossRef] - Jiang, J. Robust Model-Based Fault Diagnosis for Dynamic Systems; Kluwer Academic Publishers: Amsterdam, The Netherlands, 2002; Volume 38. [Google Scholar]
- Zhang, Y.; Jiang, J. Bibliographical Review on Reconfigurable Fault-Tolerant Control Systems. Annu. Rev. Control
**2008**, 32, 229–252. [Google Scholar] [CrossRef] - Fourlas, G.K.; Karras, G.C. A Survey on Fault Diagnosis and Fault-Tolerant Control Methods for Unmanned Aerial Vehicles †. Machines
**2021**, 9, 197. [Google Scholar] [CrossRef] - Avram, R.C.; Zhang, X.; Muse, J. Quadrotor Actuator Fault Diagnosis and Accommodation Using Nonlinear Adaptive Estimators. IEEE Trans. Control Syst. Technol.
**2017**, 25, 2219–2226. [Google Scholar] [CrossRef] - Song, Y.; He, L.; Zhang, D.; Qian, J.; Fu, J. Neuroadaptive Fault-Tolerant Control of Quadrotor UAVs: A More Affordable Solution. IEEE Trans. Neural Netw. Learn. Syst.
**2019**, 30, 1975–1983. [Google Scholar] [CrossRef] [PubMed] - Ren, X.L. Observer Design for Actuator Failure of a Quadrotor. IEEE Access
**2020**, 8, 152742–152750. [Google Scholar] [CrossRef] - Ma, H.J.; Liu, Y.; Li, T.; Yang, G.H. Nonlinear High-Gain Observer-Based Diagnosis and Compensation for Actuator and Sensor Faults in a Quadrotor Unmanned Aerial Vehicle. IEEE Trans. Ind. Inform.
**2019**, 15, 550–562. [Google Scholar] [CrossRef] - Tahri, O.; Chaumette, F. Point-Based and Region-Based Image Moments for Visual Servoing of Planar Objects. IEEE Trans. Robot.
**2005**, 21, 1116–1127. [Google Scholar] [CrossRef] [Green Version] - Haykin, S. Neural Networks and Learning Machines, 3rd ed.; Pearson: London, UK, 2011. [Google Scholar]
- Platt, J.C. Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines; Microsoft: Redmond, WA, USA, 1999. [Google Scholar]
- Jang, S.R.; Sun, C.T. Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence; Prentice-Hall: Hoboken, NJ, USA, 1997. [Google Scholar]
- Huang, G.; Zhou, H.; Ding, X.; Zhang, R. Extreme Learning Machine for Regression and Multiclass Classification. IEEE Trans. Syst. Man Cybern. B Cybern.
**2012**, 42, 513–529. [Google Scholar] [CrossRef] [Green Version] - Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] - Tang, Y.M.; Zhang, L.; Bao, G.Q.; Ren, F.J.; Pedrycz, W. Symmetric Implicational Algorithm Derived from Intuitionistic Fuzzy Entropy. Iran. J. Fuzzy Syst.
**2022**, 19, 27–44. [Google Scholar] [CrossRef] - Pounds, P.E.I. Design, Construction and Control of a Large Quadrotor Micro Air Vehicle; The Australian National University: Canberra, Australia, 2007. [Google Scholar]
- Kingma, D.P.; Ba, J.L. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015; pp. 1–15. [Google Scholar]
- Lizarralde, F.; Leite, A.C.; Hsu, L.; Costa, R.R. Adaptive Visual Servoing Scheme Free of Image Velocity Measurement for Uncertain Robot Manipulators. Automatica
**2013**, 49, 1304–1309. [Google Scholar] [CrossRef] - Reddy, S.; Panwar, L.K.; Panigrahi, B.K.; Kumar, R. Computational Intelligence for Demand Response Exchange Considering Temporal Characteristics of Load Profile via Adaptive Fuzzy Inference System. IEEE Trans. Emerg. Top. Comput. Intell.
**2018**, 2, 235–245. [Google Scholar] [CrossRef]

**Figure 1.**Classical visual servo control for robot manipulators [5].

**Figure 8.**Membership functions and the surface of the gain factor FL unit: (

**a**) $\Vert e\Vert $; (

**b**) ${\mathit{\lambda}}_{a}$; (

**c**) surface.

**Figure 9.**Result for the first case: (

**a**) Feature trajectories; (

**b**) feature errors; (

**c**) trajectory of the quadrotor; (

**d**) rotor speeds; (

**e**) RPY.

**Figure 10.**Results for the second case: (

**a**) Feature trajectories; (

**b**) feature errors; (

**c**) trajectory of the quadrotor; (

**d**) rotor speeds; (

**e**) RPY.

Parameter | Value |
---|---|

m (mass) | 4.34 kg |

I_{xx} (Diagonal inertial element) | 0.0820 kg m^{2} |

I_{yy} (Diagonal inertial element) | 0.0845 kg m^{2} |

I_{zz} (Diagonal inertial element) | 0.1377 kg m^{2} |

r (Rotor radius) | 0.165 m |

A (Rotor disc area) | 0.0855 m^{2} |

Fault Approximator | RMSE |
---|---|

NN | 0.0120 |

ELM | 0.0770 |

Linear SVM | 0.2833 |

LSTM | 0.0454 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Yuksel, T.
Fuzzy Gain-Scheduling Based Fault Tolerant Visual Servo Control of Quadrotors. *Drones* **2023**, *7*, 100.
https://doi.org/10.3390/drones7020100

**AMA Style**

Yuksel T.
Fuzzy Gain-Scheduling Based Fault Tolerant Visual Servo Control of Quadrotors. *Drones*. 2023; 7(2):100.
https://doi.org/10.3390/drones7020100

**Chicago/Turabian Style**

Yuksel, Tolga.
2023. "Fuzzy Gain-Scheduling Based Fault Tolerant Visual Servo Control of Quadrotors" *Drones* 7, no. 2: 100.
https://doi.org/10.3390/drones7020100