# Scheme of Operation for Multi-Robot Systems with Decision-Making Based on Markov Chains for Manipulation by Caged Objects

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Manipulation Scheme

## 3. Dynamic Model

## 4. Multi-Robot Tasks

#### 4.1. Equality Tasks

#### 4.1.1. Geometric Shape Formation

#### 4.1.2. Regulation Task

#### Individual

#### Cooperative

#### 4.2. Inequality Tasks

#### 4.3. Obstacle Avoidance

#### 4.4. Collision Avoidance

#### 4.5. Geometric Permissible Region

## 5. Hierarchical Quadratic Programming

Algorithm 1 Task phase |

**Require:**- State of the group ($\xi ,\dot{\xi}$), Task phase $\mathbb{S}$, $\delta t$
**Ensure:**- Control input $\tau $
Get ${\mathcal{R}}_{p}^{o}$ with (2) $\eta ={\mathcal{R}}_{p}^{o}\dot{\xi}$ Get $M,\phantom{\rule{0.222222em}{0ex}}h$ with (6) Indicate the order of the tasks: $\mathbb{S}=[{e}_{oa},{e}_{ci},\cdots ]$ Get task Jacobians: ${J}_{s}=[{J}_{oa},{J}_{ci},\cdots ]$ Get time derivative task Jacobians: ${\dot{J}}_{s}=[{\dot{J}}_{oa},{\dot{J}}_{ci},\cdots ]$ Get $\tau $ with (30) Get $\dot{\eta}$ with (6) |

## 6. Action Phases

## 7. Discrete-Time Markov Decision Agent

**Definition 1**

**Definition 2**

**Definition 3**

**Definition 4**

**Theorem 1.**

**Sketch of the Proof.**

## 8. Simulation Results

#### 8.1. Phase 1: Formation Phase

#### 8.2. Phase 2: Grip Phase

#### 8.3. Phase 3: Manipulation Phase

#### 8.4. Phase 4: Waiting Phase

#### 8.5. Variance Value and Phase Measure

**Figure 14.**Close up of the Measure from Figure 15.

#### 8.6. Comparison of DTMC and FSM in the Presence of Unexpected Failures

## 9. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

DTMC | Discrete-Time Markov Chain |

HQP | Hierarchical Quadratic Programming |

FSM | Finite-State Machine |

## References

- Ebel, H.; Eberhard, P. Non-Prehensile Cooperative Object Transportation with Omnidirectional Mobile Robots: Organization, Control, Simulation, and Experimentation. In Proceedings of the 2021 International Symposium on Multi-Robot and Multi-Agent Systems (MRS), Cambridge, UK, 4–5 November 2021; pp. 1–10. [Google Scholar] [CrossRef]
- Simões, M.A.C.; da Silva, R.M.; Nogueira, T. A Dataset Schema for Cooperative Learning from Demonstration in Multi-robot Systems. J. Intell. Robot. Syst.
**2020**, 99, 589–608. [Google Scholar] [CrossRef] [Green Version] - Lin, S.; Liu, A.; Wang, J.; Kong, X. A Review of Path-Planning Approaches for Multiple Mobile Robots. Machines
**2022**, 10, 773. [Google Scholar] [CrossRef] - Ordaz-Rivas, E.; Rodriguez-Liñan, A.; Torres-Treviño, L. Flock of Robots with Self-Cooperation for Prey-Predator Task. J. Intell. Robot. Syst.
**2021**, 101, 39. [Google Scholar] [CrossRef] - Saenz, J.; Bugarin, E.; Santibañez, V. Kinematic and Dynamic Modeling of a 4-Wheel Omnidirectional Mobile Robot Considering Actuator Dynamics. Congreso Internacional de Robótica y Computación. 2015, pp. 115–121. Available online: https://posgrado.lapaz.tecnm.mx/CIRC2015/CIRC2015.pdf (accessed on 18 March 2023).
- Li, R.; Qiao, H. A Survey of Methods and Strategies for High-Precision Robotic Grasping and Assembly Tasks—Some New Trends. IEEE/ASME Trans. Mechatronics
**2019**, 24, 2718–2732. [Google Scholar] [CrossRef] - Juang, C.F.; Lu, C.H.; Huang, C.A. Navigation of Three Cooperative Object-Transportation Robots Using a Multistage Evolutionary Fuzzy Control Approach. IEEE Trans. Cybern.
**2022**, 52, 3606–3619. [Google Scholar] [CrossRef] [PubMed] - Dai, Y.; Kim, Y.G.; Lee, D.H.; Lee, S. Symmetric caging formation for convex polygon object transportation by multiple mobile robots. In Proceedings of the 2015 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), Busan, Republic of Korea, 7–11 July 2015; pp. 595–600, ISSN: 2159-6255. [Google Scholar] [CrossRef]
- Yazıcıoğlu, Y.; Bhat, R.; Aksaray, D. Distributed Planning for Serving Cooperative Tasks with Time Windows: A Game Theoretic Approach. J. Intell. Robot. Syst.
**2021**, 103, 27. [Google Scholar] [CrossRef] - Chandarana, M.; Hughes, D.; Lewis, M.; Sycara, K.; Scherer, S. Planning and Monitoring Multi-Job Type Swarm Search and Service Missions. J. Intell. Robot. Syst.
**2021**, 101, 44. [Google Scholar] [CrossRef] - Yang, H.; Sun, L.; Li, X. A Multi-Agent Navigation Controller for Sequential Tasks Performing. In Proceedings of the 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), Honolulu, HI, USA, 31 July–4 August 2017; pp. 1680–1684. [Google Scholar] [CrossRef]
- Wu, H.; Yan, W.; Xu, Z.; Li, S.; Cheng, T.; Zhou, X. Multimodal Prediction-Based Robot Abnormal Movement Identification Under Variable Time-length Experiences. J. Intell. Robot. Syst.
**2021**, 104, 8. [Google Scholar] [CrossRef] - Luo, S.; Wu, H.; Duan, S.; Lin, Y.; Rojas, J. Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies. J. Intell. Robot. Syst.
**2021**, 101, 51. [Google Scholar] [CrossRef] - Shen, L.; Mao, P.; Fang, Q.; Wang, J. A Trajectory Tracking Approach for Aerial Manipulators Using Nonsingular Global Fast Terminal Sliding Mode and an RBF Neural Network. Machines
**2022**, 10, 1021. [Google Scholar] [CrossRef] - Pérez-Villeda, H.M.; Arechavaleta, G.; Morales-Díaz, A. Multi-vehicle coordination based on hierarchical quadratic programming. Control. Eng. Pract.
**2020**, 94, 104206. [Google Scholar] [CrossRef] - Koung, D.; Kermorgant, O.; Fantoni, I.; Belouaer, L. Cooperative Multi-Robot Object Transportation System Based on Hierarchical Quadratic Programming. IEEE Robot. Autom. Lett.
**2021**, 6, 6466–6472. [Google Scholar] [CrossRef] - Li, H.; Ding, X. Adaptive and intelligent robot task planning for home service: A review. Eng. Appl. Artif. Intell.
**2023**, 117, 105618. [Google Scholar] [CrossRef] - Obregón-Flores, J.; Arechavaleta, G.; Becerra, H.M.; Morales-Díaz, A. Predefined-Time Robust Hierarchical Inverse Dynamics on Torque-Controlled Redundant Manipulators. IEEE Trans. Robot.
**2021**, 37, 962–978. [Google Scholar] [CrossRef] - Alam, T.; Rahman, M.M.; Carrillo, P.; Bobadilla, L.; Rapp, B. Stochastic Multi-Robot Patrolling with Limited Visibility. J. Intell. Robot. Syst.
**2020**, 97, 411–429. [Google Scholar] [CrossRef] - Mirzaei, F.; Pouyan, A.A.; Biglari, M. Automatic Controller Code Generation for Swarm Robotics Using Probabilistic Timed Supervisory Control Theory (ptSCT). J. Intell. Robot. Syst.
**2020**, 100, 729–750. [Google Scholar] [CrossRef] - Nguyen, V.; Kim, O.T.T.; Pham, C.; Oo, T.Z.; Tran, N.H.; Hong, C.S.; Huh, E.N. A Survey on Adaptive Multi-Channel MAC Protocols in VANETs Using Markov Models. IEEE Access
**2018**, 6, 16493–16514. [Google Scholar] [CrossRef] - Meyn, S.P.; Tweedie, R.L. Markov Chains and Stochastic Stability; Springer London: London, UK, 1993. [Google Scholar] [CrossRef]
- Poznyak, A.S.; Alazki, H.; Soliman, H.M. Invariant-set design of observer-based robust control for power systems under stochastic topology and parameters changes. Int. J. Electr. Power Energy Syst.
**2021**, 131, 107112. [Google Scholar] [CrossRef] - Yang, Y.; Xi, X.; Miao, S.; Wu, J. Event-triggered output feedback containment control for a class of stochastic nonlinear multi-agent systems. Appl. Math. Comput.
**2022**, 418, 126817. [Google Scholar] [CrossRef] - Han, Y.; Zhou, S. Novel Criteria of Stochastic Stability for Discrete-Time Markovian Jump Singular Systems via Supermartingale Approach. IEEE Trans. Autom. Control
**2022**, 67, 6940–6947. [Google Scholar] [CrossRef] - Qi, T.; Chen, J.; Su, W.; Fu, M. Control Under Stochastic Multiplicative Uncertainties: Part I, Fundamental Conditions of Stabilizability. IEEE Trans. Autom. Control
**2017**, 62, 1269–1284. [Google Scholar] [CrossRef] - Long, S.; Zhong, S. Mean-square exponential stability for a class of discrete-time nonlinear singular Markovian jump systems with time-varying delay. J. Frankl. Inst.
**2014**, 351, 4688–4723. [Google Scholar] [CrossRef] - Francelino, E.; Pereira, M.; Inoue, R.; Terra, M.; Siqueira, A.; Nogueira, S. Markov System with Self-Aligning Joint Constraint to Estimate Attitude and Joint Angles between Two Consecutive Segments. J. Intell. Robot. Syst.
**2022**, 104, 43. [Google Scholar] [CrossRef] - Cao, X.; Jain, P.; Goodrich, M.A. Adapted Metrics for Measuring Competency and Resilience for Autonomous Robot Systems in Discrete Time Markov Chains. In Proceedings of the 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic, 9–12 October 2022; pp. 71–76. [Google Scholar] [CrossRef]
- Ralph-Christoph, W.; Bellenberg, M.; Schwarzenberger, D. Festo’s Robotino User Guide; Manual Order No.:KG 544305; FESTO: Esslingen, Germany, 2009. [Google Scholar]
- Lang, A.; Petersson, A.; Thalhammer, A. Mean-square stability analysis of approximations of stochastic differential equations in infinite dimensions. BIT Numer. Math.
**2017**, 57, 963–990. [Google Scholar] [CrossRef] [Green Version]

**Figure 1.**The Manipulation scheme involves six interconnected blocks: 1. Robots, representing the omnidirectional robots; 2. Environment, the operational space for the robots; 3. Tasks, the specific goals to be achieved by the robots; 4. Phases, the stages involved in the manipulation scheme; 5. DTMC, which utilizes a discrete-time Markov chain; and 6. HQP, which computes the control of the robots to carry out simultaneous tasks.

O | Inertial Frame |

P | Referential in the robot C.o.M. |

n | number of robots |

$\xi $ | robot extended state |

${q}_{i}$ | i-robot state |

${x}_{i}$ | i-robot position along x-axis |

${y}_{i}$ | i-robot position along y-axis |

${\theta}_{i}$ | i-robot orientation along z-axis |

${R}_{O}^{P}\left(\theta \right)$ | Rotation Matrix from Frame O to P |

${\mathcal{R}}_{O}^{P}\left(\theta \right)$ | Extended Rotation Matrix |

${M}_{i}$ | i-robot inertia Matrix |

${M}_{\varphi i}$ | i-robot wheels inertia Matrix |

${h}_{i}$ | non-linear Vector from i-robot |

${\tau}_{i}$ | i-robot control input |

${\tau}_{\varphi i}$ | i-robot control input for each wheel |

M | Extended inertia Matrix |

h | non-linear extended vector from i-robot |

${e}_{Q}$ | Q-task error |

${J}_{Q}$ | Q-task Jacobian |

${\dot{J}}_{Q}$ | Time derivative from ${J}_{Q}$ |

${u}_{Q}$ | Q-task auxiliary control |

$\mathbb{S}$ | Behaviour state space |

${\mathbb{S}}_{j}$ | j-state of behaviour |

${\alpha}_{j}$ | j-state distribution |

Phase | Task | Error | Hierarchy |
---|---|---|---|

Phase 1 Formation | Obstacle avoidance | ${e}_{oa}$ | 1 |

Collision avoidance | ${e}_{om}$ | 2 | |

Geometric formation | ${e}_{ci}$ | 3 | |

Phase 2 Grip | Obstacle avoidance | ${e}_{oa}$ | 1 |

Collision avoidance (equidistant) | ${e}_{om}$ | 2 | |

Geometric formation | ${e}_{ci}$ | 3 | |

Phase 3 Manipulation | Obstacle avoidance | ${e}_{oa}$ | 1 |

Collision avoidance | ${e}_{om}$ | 2 | |

Permissible space | ${e}_{pi}$ | 3 | |

Cooperative regulation | ${e}_{r}$ | 4 | |

Phase 4 Waiting | Obstacle avoidance | ${e}_{oa}$ | 1 |

Collision avoidance | ${e}_{om}$ | 2 | |

Individual regulation | ${e}_{ri}$ | 3 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Arreguín-Jasso, D.; Sanchez-Orta, A.; Alazki, H.
Scheme of Operation for Multi-Robot Systems with Decision-Making Based on Markov Chains for Manipulation by Caged Objects. *Machines* **2023**, *11*, 442.
https://doi.org/10.3390/machines11040442

**AMA Style**

Arreguín-Jasso D, Sanchez-Orta A, Alazki H.
Scheme of Operation for Multi-Robot Systems with Decision-Making Based on Markov Chains for Manipulation by Caged Objects. *Machines*. 2023; 11(4):442.
https://doi.org/10.3390/machines11040442

**Chicago/Turabian Style**

Arreguín-Jasso, Daniel, Anand Sanchez-Orta, and Hussain Alazki.
2023. "Scheme of Operation for Multi-Robot Systems with Decision-Making Based on Markov Chains for Manipulation by Caged Objects" *Machines* 11, no. 4: 442.
https://doi.org/10.3390/machines11040442