Toward Adaptive Human–Robot Collaboration for the Inclusion of People with Disabilities in Manual Labor Tasks

Abstract

While human–robot collaboration is already integrated in industrial and service robotics applications, it is only used with able-bodied workers. However, collaboration through assistive robots is a major driver toward the inclusion of people with disabilities, which was demonstrated in recent research projects. Currently, inclusive robot workplaces have to be customized toward the work process and the individual needs of the person. Within, robots act along a fixed schedule and are not able to adapt to changes within the process or the needs of the interacting person. Hence, such workplaces are expensive and unappealing for companies of the first labor market, and do not realize the full potential of the technology. In this work, we propose a generalized approach toward the inclusion of people with disabilities with collaborative robots. To this end, we propose a system that analyzes the in situ capabilities of a person using a two-stage reasoning approach. The methodology is based on an ontology that allows the matchmaking of individual capabilities with process requirements. Capabilities are modeled in two time frames, through which fast (e.g., fatigue) and slow effects (e.g., worsening of illness) become distinguishable. The matchmaking is used in task allocation to establish high-level control over the assistive system. By this approach, inclusive workplaces become autonomously adaptive to the in situ capabilities of the individual person, without the need for customization. Therefore, collaborative workplaces become not only inclusive, but a contributor toward a labor market for all.

1. Introduction

While human–robot collaboration (HRC) in general is a well-studied field, it is often applied to sectors in which able-bodied workers are supported to reduce strain and increase their ergonomics, besides other benefits. However, those general approaches cannot be used to include people with disabilities (PwD; also used as “person with disabilities” in this work), as their capabilities vastly differ from those of the able-bodied worker. HRC is a tool particularly well suited to assist PwD in the first labor market and manual labor tasks. In Gemany, PwD often work in workshops, which are isolated from the regular labor market and only offer simple tasks. Therefore, workshops essentially establish a parallel labor market, i.e., distinction in the “first labor market” and others.

PwD need to be treated individually, as their type of particular disability is often a combination of different partial disabilities. This results in a highly individual capability profile. If HRC is to be used to include PwD in manual labor tasks, the robot has to adapt to the person’s individual needs. However, to keep HRC attractive to the company, customization of workplaces is a non-tolerable socio-economical risk, as this raises the costs and turns inclusion into a financial risk. Therefore, our effort is to design a general approach toward HRC that allows the robot to adapt directly to the observed capability profile of the user rather than to a predefined set of capabilities. The technology is intended for use in manufacturing and implements capabilities required in such applications. In this work, we propose the following:

• A matchmaking ontology that models the collaborative process, including sub-ontologies that model the capability profiles and the process steps, and a sub-ontology that maps sensor observations to capability qualifications.
• A system design that allows reasoning on the evolution of the capability profiles and which interfaces with automated task allocation.

Note that a methodology tailored for PwD is also suitable for other user groups. First, from a modeling viewpoint, able-bodied workers are PwD without limited capabilities. Second, the user group of PwD comprises not only people which are permanently disabled, but also the elderly, temporarily disabled people (e.g., due to illness or injury), or people new to the working process. Hence, every person is characterized by a set of (partially) limited or non-limited capabilities (compare Section 3.2) and may be modeled with the proposed ontologies, and, therefore, be supported by a robot in the HRC process.

This work describes the initial steps taken toward adaptive HRC for PwD. We propose a novel methodology, which is currently in a descriptive state but will be validated over the course of the next years. The paper is structured as follows: First, we discuss related work in Section 2. Section 3 presents the system design and modeling of the capability profiles designed for PwD. Section 4 discusses the composition of the sub-ontologies used in the robotic assistance matchmaking ontology (RAMO), including the profile ontology (Section 4.1), the process ontology (Section 4.2), and the observable capability emissions ontology (OCEO; Section 4.3). Section 5 describes how the ontologies are embedded in automated task allocation. Section 6 discusses the proposed system design and gives an outlook toward future research. Finally, Section 7 summarizes the work.

2. Related Work

For the design of inclusive workplaces in general, as an initial step, a comparison of the individual capability profile with the process requirements is necessary. The comparison allows to adapt the workplace according to the person’s individual needs with the best possible support for the PwD. This section first introduces some concepts to perform capability matchmaking for PwD (Section 2.1). Subsequently, related work in knowledge representation is discussed (Section 2.2), which is later used to model information to assess the PwDs’ capabilities.

2.1. Capability Matchmaking for People with Disabilities

IMBA (from German: “Integration von Menschen mit Behinderungen in die Arbeitswelt”) is a tool to compare and document the human capabilities and the workplace requirements, established by the German Ministry of Health and Social Security [1,2]. The documented capabilities are elicited by means of an occupational health examination. However, IMBA is not a survey instrument in the sense of a test procedure, but rather a documentation method with general assessment aids. Therefore, IMBA does not allow a clear quantitative survey in the assessment of process requirements. Ranz et al. [3] decompose industrial processes on the basis of a directed graph into a sequence of work processes, which in turn can be described as a sequence of fundamental motions. In industry, the method–time measurement (MTM) [4] approach is a common way to model manual work processes using fundamental motions. The primary method MTM-1 [5] consists of 19 fundamental motions, which are extended by skills such as hearing, seeing, calculating, and reading.

The tool RAMB (“robotic assistance for manufacturing including people with disabilities”), introduced in [6] and elaborated in [7], is used to identify those process steps in which the PwD require individual assistance. This is achieved by combining the process decomposition according to MTM and IMBA such that the process requirements can be evaluated uniformly and compared with the capability profile of a PwD. The profile representation in RAMB is discussed in Section 3.2.

2.2. Knowledge Representation

For advanced AI and HRC applications, often many different sensors have to be used. The vast amount of sensor data has to be evaluated. These evaluations, again, have to be related to each other and thus re-evaluated. Furthermore, the knowledge has to be exchanged between humans and robots or other artificial systems. This requires the development of complex knowledge representation systems that can collect and combine data from different sources and integrate them in a meaningful knowledge base which has a common conceptualization for all users as indicated by Prestes et al. [8].

The term ontology originates from philosophy and denotes a systematic representation of existence. However, in computer science, the term is used as a formal conceptualization of a domain of knowledge. That means describing the universe of discourse ontology by a set of definitions, e.g., classes, relations, constraining axioms, properties, and their instances [9]. Ontologies gained special importance through the approach of the Semantic Web [10], in which the World Wide Web should become computer readable through semantic metadata. Meanwhile, ontologies are used in many applications of robotics as complex and expressive knowledge representation systems to improve the autonomy of robots by enabling fast and convenient reasoning [11].

Projects such as the OpenRobots Ontology (ORO) [12] or KnowRob [13] are examples of specific knowledge base representations for robots. For example, KnowRob and its successor KnowRob2 [14] are knowledge processing systems for autonomous personal robots that are to perform everyday manipulation tasks. Their ontologies consist of encyclopedic knowledge of the task domain and general knowledge about the robot’s environment, action models, instances, and computables. Action models describe possible manipulations that can be performed, and computables are used for creating instances or relations between instances. The ORO [12] focuses on human–robot interaction needs and implements a fast, standard-based knowledge storage, where different perception modules, users, or reasoners can pull or push needed or inferred knowledge.

3. Toward Adaptive and Inclusive Human–Robot Collaboration

To allow robots to adapt to the PwDs’ needs, we base our approach on the sense–plan–act principle. The robot observes its environment using exteroceptive sensors, e.g., cameras and force–torque sensors. The sensor data are processed to gather information on the human behavior. This information is then used to reason on the human capabilities and perform matchmaking regarding the process requirements. The matchmaking is used in a task scheduler to allocate tasks between the human and the robot. In the following section, we present the system design and capability profiling used to this end. Following definitions are used and detailed accordingly:

Generic
T	Task
$S{P}_l$	Standard process
$\mathbf{g}$	Set of features
$g_j$	Feature, $g_j\in \mathbf{g}$
$\mathbf{p}$	Capability profile
$f_i$	Capability, $f_i\in \mathbf{p}$
${\mathbf{b}}_k$	Basic element, ${\mathbf{b}}_k\subseteq \mathbf{p}$
Qualified
${\mathbf{p}}^i$	Capability profile qualified for a person i
$f_i^i$	Capability qualified for a person i, $f_i^i\in {\mathbf{p}}^i$
${\mathbf{b}}_{k,l}$	Basic element ${\mathbf{b}}_k$ qualified in a standard process $S{P}_l$

3.1. System Design

The system consists of sequenced modules interfacing with a database (see Figure 1). Sensor data are processed to generate features $g_j$ on the system state. Such features are, for example, the relation between skeleton joints or the force transmitted into the robot structure. The set of features $\mathbf{g}=\{g_0,g_1,\dots ,g_n\}$ is post-processed in a reasoning module to translate features into the human capability profile $\mathbf{p}$ consisting of individual capabilities $f_i$. Individual features in the feature set may not be available in situ. Hence, some features may be deactivated for pre-reasoning.

The output of the pre-reasoning is the so-called anytime profile ${\mathbf{p}}_{any}$ (see Section 3.3.2) that is compared to the process requirements in matchmaking. The decision of whether the human can fulfill the task is then forwarded to the task allocation, which generates a plan of actions on the shared working process. To this end, the task planner manages standard task durations in the database. The task schedule is then forwarded to robot control and the human in an accessible way, e.g., using visual or auditory feedback. The task planning module reasons internally on the success state of the performed actions by incorporating environmental information and the standard task durations. Based on this, new information is gathered that is then used for post-reasoning. The latter is mainly used to update the so-called offline profile ${\mathbf{p}}_{off}$ (see Section 3.3.1), which represents an ex situ forecast of the person’s capabilities throughout the day, and which is used as initialization of ${\mathbf{p}}_{any}$ at the day’s start.

3.2. Capability Modeling and Matchmaking for People with Disabilities

In [6], capabilities are defined in relation to a work task T. A task consists of a set of standard processes $S{P}_l$, which again define a sequence of basic elements $b_k\subseteq \mathbf{p}$, e.g., grasp, move. Hereby, index l denotes the standard process and k the specific basic element, which are mutually independent. The qualification of basic elements is set according to a standard process; hence, $b_{k,l}$ denotes the qualification of $b_k$ in $S{P}_l$. Further, $\mathbf{p}=\{f_1,f_2,\dots ,f_m\}$ denotes the set of all assessable capabilities ($m=87$ in RAMB, see

Table 1. 87 capabilities defined in RAMB, structured in six main categories: ergonomics, motor functions, perception, cognitive, body motion, and complex capabilities (d.—double sided; s.—single sided).

Cat.	ID	Capability	Cat.	ID	Capability	Cat.	ID	Capability
<ergonomics>	Posture		<motorfunction>	Hand/Fingers		<bodymotion>	Walking/Climbing
	01	Sitting		31	Encompassing handle,		60	Walking, at level
	02	Standing			fist closure, d.		61	Walking, on inclined plane
	03	Kneeling		32	Encompassing handle,		62	Walking, on loose/
	04	Squatting			fist closure, s			uneven ground
	Inclined/Stooped			33	Encompassing handle,		63	Ascending
	05	Sitting, inclined			hand closure, d.		64	Climbing
	06	Sitting, bent		34	Encompassing handle,		65	Crawling/Sliding
	07	Standing, inclined			hand closure, s.	<complex>	Lifting
	08	Standing, bent		35	Hand, contact handle, d.		66	Horizontal
	Arms restrained			36	Hand, contact handle, s.		67	Floor to waist height
	09	Sitting/Standing,		37	Hand rotation, d.		68	Waist to eye height
		arms frontwards		38	Hand rotation, s.		69	Waist to overhead height
	10	Sitting/Standing,		39	Finger, grasping handle, d.		Carrying
		arms overhead		40	Finger, grasping handle, s.		70	Sideways
	11	Supine, arms overhead		41	Finger, contact handle, d.		71	Front of body
	12	Lateral, arms frontwards		42	Finger, contact handle, s.		72	On the back
<motorfunction>	Head/Neck			43	Thumb, contact handle, d.		Physique
	13	Rotation		44	Thumb, contact handle, s.		73	Pushing (objects)
	14	Bent, lateral	<perception>	Sight			74	Dragging (objects)
	15	Bent, sideways		45	Visual acuity, close		75	Physical stamina
	Torso			46	Visual acuity, far		76	Balance
	16	Rotation, sit		47	Spatial vision		Fine motor skills
	17	Rotation, stand		48	Field of view		77	Hand dexterity, d.
	18	Bent/Erect		49	Color vision		78	Hand dexterity, s.
	Leg/Foot			50	Mesopic vision		79	Finger dexterity, d.
	19	Squat		Hearing			80	Finger dexterity, s.
	20	Pedal actuation		51	Listening comprehension		81	Hand-arm stability
	Arm			52	Noise/Speech pattern		82	Control accuracy
	21	Reaching, over shoulder, d.			recognition		83	Coordination of
	22	Reaching, over shoulder, s.		53	Frequency			multiple limbs
	23	Reaching, frontwards, d.		54	Volume		84	Wrist speed
	24	Reaching, frontwards, s.		55	Direction		85	Finger speed
	25	Reaching, sidewards, d.	<cognitive>	Basic education			86	Movement speed, arms
	26	Reaching, sidewards, s.		56	Vocal output/Speaking		87	Movement speed, legs
	27	Reaching, backwards, d.		57	Reading comprehension
	28	Reaching, backwards, s.		58	Calculating
	29	Forearm rotation, d.		59	Writing
	30	Forearm rotation, s.

), whereas each capability has a qualification $f_i^i=[-3,3]\subset \mathbb{Z}$. The qualified capabilities are aggregated in the qualified profile $p^i$, which is user dependent. After qualification, a regular capability is defined by $f_i^i\ge 0$, and $f_i^i<0$ denotes a partial disability. While the set of capabilities in each basic element does not change, their qualification $b_{k,l}$ may differ between standard processes containing them, and hence we have the following:

$$\left|\right|{\mathbf{b}}_{k,1}-{\mathbf{b}}_{k,2}\left|\right|\ge 0.$$

(1)

Hereby, the Manhattan metric is a well-suited choice, as it directly indicates the number of deviations.

To perform matchmaking, the qualified basic elements are compared to the qualified capabilities of the human ${\mathbf{p}}^i$, given by

$$\mathsf{\Delta }{\mathbf{b}}_{k,l}^i={\mathbf{b}}_{k,l}-({\mathbf{p}}^i\cap {\mathbf{b}}_k).$$

(2)

If any entry in $\mathsf{\Delta }{\mathbf{b}}_{k,l}^i$ is positive, the specific basic element in the standard process cannot be performed by the human and, consequently, needs to be allocated to the robot. Note that superscript i denotes the capabilities of a specific human and the derived qualifications, and that a basic element is qualified by the standard process and not by the human. Only the difference in matchmaking is qualified by the human and the standard process; hence, $\mathsf{\Delta }{\mathbf{b}}_{k,l}^i$ carries the corresponding indexes (k: basic element, l: standard process) and the superscript i.

3.3. Dynamics of Capability Profiles

In reality, a capability profile is non-static and changes over time. In [7], this effect is modeled in a Langevin system. The qualified profile can, therefore, be modeled as

$${\mathbf{p}}^i\left(t\right)={\tilde{\mathbf{p}}}^i\left(t\right)+{\mathbf{ϵ}}^i\left(t\right),$$

(3)

where ${\tilde{\mathbf{p}}}^i\left(t\right)$ are the qualified capabilities originating from the individual disability complex, and where ${\mathbf{ϵ}}^i\left(t\right)$ are small fluctuations in the capabilities. The origin of these capabilities’ time dependency is significantly different. While ${\tilde{\mathbf{p}}}^i\left(t\right)$ changes only slowly, it is mostly superimposed by the fast variations in ${\mathbf{ϵ}}^i\left(t\right)$, which is typical in a Langevin system.

To transfer this idea to manual labor tasks, two effects have to be taken into account. First, the worker arrives with a similar capability profile at the start of the day. Second, the capabilities deteriorate over the course of the day. To model this property, two time concepts are introduced: the work time $0\ge \tau$ (which is delimited by a shift’s end) and the global time $t_0\ge 0$, which is fixated at the start of a day or shift; hence, $t=t_0+\tau$. Effectively, $t_0$ becomes constant when observed in the time frame of $\tau$. Therefore, we assume that the influence of the disability changes so slowly that it is assumed constant over the course of the day. Therefore, Equation (3) is rewritten as

$${\mathbf{p}}^i\left(t\right):={\mathbf{p}}^i(t_0,\tau )={\tilde{\mathbf{p}}}^i\left(t_0\right)+{\mathbf{ϵ}}^i\left(\tau \right).$$

(4)

In adaptive HRC, two types of profiles are managed: (1) the offline profile that establishes a baseline to the matchmaking on a per-day basis, and (2) the anytime profile that reflects the in situ capabilities of the person.

3.3.1. Offline Profile

The offline profile is defined as the global time-dependent qualified capability vector

$${\mathbf{p}}_{off}^i\left(t\right)={\tilde{\mathbf{p}}}^i\left(t_0\right)+\mathsf{\Delta }{\tilde{\mathbf{p}}}^i\left(\tau \right)+\mathbf{\nu }\left(t\right),$$

(5)

where $\mathsf{\Delta }{\tilde{\mathbf{p}}}^i\left(\tau \right)$ is the daily fluctuation, and $\mathbf{\nu }\left(t\right)$ is a noise vector in which entries are independent. The offline profile is initialized with $\mathsf{\Delta }{\tilde{\mathbf{p}}}^i(\tau =0)=\mathbf{0}$, and ${\tilde{\mathbf{p}}}^i(t_0=0)$ incorporating the data from a medical preliminary examination (MPE) as usually performed in advance of including PwD in work processes. As this—and all sensor data later in the process—is subject to noise or personal bias (medical personnel), $\mathbf{\nu }\left(t\right)$ is introduced in Equation (5).

With the MPE, ${\mathbf{p}}_{off}^i\left(t\right)$ is constant over the course of a day. To better predict the fluctuations during the day and, therefore, decrease the size of the fluctuations in $\mathbf{\nu }\left(t\right)$, $\mathsf{\Delta }{\tilde{\mathbf{p}}}^i\left(\tau \right)$ is re-evaluated in two ways: (1) continuously by post-reasoning, and (2) by the mining of past profiles and fitting $\mathsf{\Delta }{\tilde{\mathbf{p}}}^i\left(\tau \right)$ accordingly. The latter establishes a global optimization problem with the aim of reducing $\mathbf{\nu }\left(t\right)$ to a minimum, and hence,

$$\underset{t->\infty }{lim}\left|\right|\mathbf{\nu }\left(t\right)\left|\right|=0$$

(6)

In states close to initialization, the values in $\mathbf{\nu }\left(t\right)$ will superimpose the other factors, particularly $\mathsf{\Delta }{\tilde{\mathbf{p}}}^i\left(\tau \right)$ (see Figure 2).

3.3.2. Anytime Profile

The anytime profile is defined as the in situ qualified capability vector

$${\mathbf{p}}_{any}^i\left(t\right)={\mathbf{p}}_{off}^i\left(t\right)+\mathbf{\eta }.$$

(7)

The anytime profile is the main output of the reasoning modules and represents the in situ capabilities of the person superimposed by stochastic system noise $\mathbf{\eta }$, which is mainly influenced by the following:

• Noise in the sensor signal.
• Uncertainties in feature extraction.
• Propagated uncertainties from hidden Markov model (HMM; see Section 4.3).
• Uncertainties in the modeling approach.

As uncertainties in the model and HMM are Gaussian, and sensor noise is assumed white noise, $\mathbf{\eta }$ may be modeled as Gaussian noise.

4. Ontology-Based Reasoning

The next step toward automatic task allocation and scheduling is the estimation of a person’s in situ capabilities. As mentioned in Section 3.1, sensor data are processed into features $g_j$ representing certain information on the human behavior, the environment, and the interaction between human and robot. For data processing, typical methods are used, e.g., OpenPose [15] for skeleton tracking or the method by Buondonno and De Luca [16] for interaction force computation. The explicit origin of processed sensor data is not relevant, as the methodology discussed in this section establishes a generalist approach to transform these data into a capability profile estimate.

Equation (2) establishes a common base at the human’s capabilities to perform capability matchmaking onto process requirements. There are two ways to reach these capabilities $f_i$: from the sensors, and from the task defined by the work process. Therefore, there exists an open loop from the processed sensor data to the task. This dependency is modeled in the novel robotic assistance matchmaking ontology (RAMO) that is depicted in Figure 3.

RAMO consists of two sub-ontologies. First, the observable capability emissions ontology (OCEO) models the emissions of the qualified capabilities measurable by exteroceptive sensors. Second, the process ontology as used in RAMB [7] models the tasks and their structural components. Both sub-ontologies are connected by the profile ontology defined by the capabilities $f_i$.

4.1. Profile Ontology

The profile ontology models the relations and influences between the capabilities $f_i$ of a human, which are accumulated in the profile $\mathbf{p}$. Capabilities are structured in six main categories: ergonomics, motor functions, perception, cognitive, body motion, and complex capabilities. Usually, human capabilities are not mutually independent, e.g., the capability to perform a squat ($f_{19}$) also influences the capability of climbing ($f_{64}$). Particularly, complex capabilities, e.g., horizontal lifting ($f_{66}$), are highly dependent on basic capabilities, e.g., reaching sideways ($f_{25}$) or torso rotation ($f_{16}$ and $f_{17}$).

Relations between capabilities are modeled directed and may have minor, major, or no influence on the other capabilities. A directed relation is chosen, as some capabilities may be influenced by multiple other capabilities, but the cause of the qualification may be manifold. For example, an impaired capability reaching sideways may either come from an impairment in the limbs or the torso, which manifests in different capabilities. Hence, this example is modeled as depicted in Figure 4. Capabilities and their relations are modeled based on IMBA [1,2] and according to observations from user tests during the Next Generation https://www.nextgeneration-mrk.de/ (accessed on 22 February 2023) project.

4.2. Process Ontology

The process ontology models the structure explained in Section 3.2 (compare right-hand side in Figure 3). A work process is defined as a task T. This task consists of standard processes $S{P}_l$. Standard processes are characterized by entry and end states. These are defined with the aim to reduce process dependencies between standard processes, i.e., all sequenced basic elements should be accumulated in one $S{P}_l$. Therefore, standard processes are interchangeable if the real system fulfills the requirements of their entry state. A standard process then consists of basic elements $b_{k,l}$ that are sequenced in a fixed manner and must not be interchanged. However, the basic elements can still be inaccessible to the human and, therefore, the robot needs to assist or take them on completely. It is subject to the task planner to assign basic elements to both agents or solely to the robot. There might occur the situation that the human can fulfill only a small margin of basic elements in a $S{P}_l$ and in which it is more efficient for the robot to take on the complete $S{P}_l$.

4.3. Observable Capability Emissions Ontology

The observable capability emissions ontology (OCEO) is based on the assumption that qualifications of capabilities manifest in features observable by sensors, system states, and meta information (compare Figure 3). Sensor data are processed into features characterizing the human behavior, the environment, and other agents. System states represent the continuous evaluation of successfully fulfilled actions in the task planner (see Section 5) and meta data are a composition of non-sensor data from outside the system, e.g., the information collected in a medical examination. Sensor, meta, and system data form the input layer. The features are then fed into a multilevel hidden Markov model (compare, for example, [17]). In a HMM, non-observable states cause so-called emissions, which are observable and deliver evidence on the origin of the emission’s cause, e.g., impaired vision may result in an unsteady gait. The unsteady gait may be measured by a skeleton model and, therefore, may be used as an emission in the HMM. In a multilevel approach, emissions exist that cause emissions themselves, hence obscuring the capability qualifications over multiple layers. Transitions in and between the hidden Markov layers are annotated with their uncertainties, which are propagated toward the matchmaking layer.

It is to be noted that some features are directly measurable (e.g., $f_1$ in Figure 3), while others (e.g., $f_4$ in Figure 3) are not observable at all. Further, features or emissions do not directly qualify a capability. Instead, they map onto a qualification value in the range $[-3,3]\subset \mathbb{Z}$, whereas not all qualification values may be connected to the same emission or feature. Indeed, some qualification values may only manifest in very specific emissions, e.g., a minor vision impairment ($f_{46}$) may result in unsteady gait, but a major vision impairment may also result in tripping or collisions with the environment. The example is detailed in Figure 5.

In the matchmaking layer, the qualified capability profile is compared with the qualified basic elements. Hence, a decision is made as to whether a basic element may be allocated to the human. To this end, the propagated uncertainties are analyzed and reasoned on. While the matchmaking layer itself is not part of either the OCEO or the process ontology, both sub-ontologies end in the capability profile. Both sub-ontologies and the matchmaking layer generate the robotic assistance matchmaking ontology RAMO.

5. AI Task Planning for Human-Robot-Teams

Automated task planning or AI task planning methods compute the actions that must be executed by agents to bring the system to a desired goal state. The goal state of the methodology described in this work is the fulfillment of task T. AI task planning methods (e.g., [18]) perform two steps: (1) select, order, and instantiate abstractly defined actions to specific actions, and (2) optimize the preliminary plan with respect to a set of criteria. The execution of specific actions brings the system from a given initial state to a given goal state, e.g., the generic action grasp agent object place can be instantiated to grasp robot screw table (read: “the robot grasps the specific screw at the specific table”). This action is assigned to the robot agent. The ordering and instantiation process of actions considers the dependencies between these actions, e.g., a place action is selected only when an agent handles an object, and hence, beforehand, a grasp action has to be performed. This is also depicted in the standard processes, which define a sequence of basic elements. Note that the task planner may either allocate individual basic elements ${\mathbf{b}}_{k,l}$ or the whole standard process $S{P}_l$ to the individual agent. The result of the first solving step is a preliminary plan containing ordered and instantiated actions $a_i$, whereas $a_i$ is the robot-readable abstraction of the basic elements ${\mathbf{b}}_{k,l}$. In a second step, AI task planning methods optimize the preliminary plan with respect to a set of criteria (e.g., time). The methods adapt the start times and the order of the planned actions, without violating the dependencies between them, to obtain a plan with minimal makespan (execution time). The final result is the plan

$$\pi =<a_0,\dots ,a_n>.$$

(8)

The advantages of AI task planning methods are that they require only abstractly defined actions (analogous to ${\mathbf{b}}_k$), an initial planning state, and a set of goals that must hold in a goal state for solving a planning problem [19]. AI task planning methods can be used as flexible, high-level control strategies. They are able to generate new plans for manifold planning situations, e.g., when new orders arrive or when the characteristics of the agents change. The latter is particularly important when the capabilities of an agent are expected to be dynamically adaptable, which is the case for PwD. For each new planning situation, only the initial and the goal states must be re-set, while the remaining definitions and the planning process itself must not be further adapted. AI task planning approaches were already successfully deployed in several of standard HRC applications using the ROSPlan (https://kcl-planning.github.io/ROSPlan/ (accessed on 22 February 2023)) framework [20,21,22] and may be adapted toward the methodology described in this work.

The capability profiles ${\mathbf{p}}_{any}^i\left(t\right)$ are encoded as preconditions for the defined generic actions of a planning problem (see Algorithm 1).

Algorithm 1 Definition of a generic action, of an initial state, and of a goal state.

1: $(:\mathrm{action}\mathrm{grasp}$  2: $:\mathrm{parameters}(?\mathrm{r}-\mathrm{robot}?\mathrm{i}-\mathrm{item}?\mathrm{f}1,\mathrm{f}2-\mathrm{capability})$  3: $:\mathrm{precondition}(\mathrm{and}(\mathrm{near}?\mathrm{r}?\mathrm{i})(?\mathrm{f}1<=3)(?\mathrm{f}2>0\left)\right)$  4: $:\mathrm{effect}(\mathrm{and}(\mathrm{item}\_\mathrm{grasped}(?\mathrm{r}?\mathrm{i}\left)\right)\left)\right)$  5: −−−  6: $(:\mathrm{objects}\mathrm{o}1-\mathrm{item}\mathrm{r}1-\mathrm{robot})$  7: −−−  8: $(:\mathrm{init}(=\mathrm{f}1-2)(=\mathrm{f}22\left)\right)$  9: −−−  10: $(:\mathrm{goal}(\mathrm{and}\mathrm{item}\_\mathrm{grasped}(\mathrm{r}1,\mathrm{0}1)\left)\right)$

Further, these profiles are instantiated to specific values as part of the initial state of a planning problem, analogous to the specification in Section 3.2. In Algorithm 1, the grasp action has the constraint $f_2>0$ as one of its preconditions, i.e., this action can be planned and assigned to a human only if in the initial state of the planning problem, the standing capability $f_2^i$ is instantiated to a value $>0$, e.g., to the value $f_2^i=3$. The solving process automatically assigns all planned actions to agents by taking into consideration the capability profiles ${\mathbf{p}}_{any}^i\left(t\right)$ that are defined in relation to a work task T.

As described in Section 3.3, the capability profiles are dynamic. The allocation of tasks to the agents (robots or PwD) must consider these changes. AI task planning methods enable a seamless integration of these changes for the planning process. Solely the initial state of the planning problems must be adapted, e.g., the considered capability can be modified for a new planning problem from $f_2^i=2$ to $f_2^i=-2$. This is the only modification required by the planning system, such that it can determine new plans that consider the new profiles.

The generated plan $\pi$ must be executed such that the application reaches the targeted goal state in the real world. The AI task planning framework ROSPlan enables a combined planning and execution approach [23]. It uses action interface modules to send execution commands to the agents (robots or PwD) and to supervise the execution of the dispatched actions. As part of this work, the existing AI task planning framework is extended with further modules to allow system states as input to the reasoning modules (see $g_5$ in Figure 3). The new modules translate the execution state (i.e., success or failure) of the actions to features that can be integrated with the reasoning modules. These again transport important knowledge for the assessment of the capability profile ${\mathbf{p}}_{any}^i\left(t\right)$ (see transition to post-reasoning in Figure 1). In this way, capability profiles ${\mathbf{p}}_{any}^i\left(t\right)$ and ${\mathbf{p}}_{off}^i\left(t\right)$, respectively, are additionally updated based on the execution success of the actions and not only based on sensor data. This also closes the control loop from the task execution toward the initial capability assessment.

6. Discussion and Outlook

As mentioned already in Section 1, the proposed methodology is only a system design and ontology description that has not yet been implemented in real work processes. Therefore, this section discusses some aspects that need to be considered when deploying the proposed methodology.

Due to its novel character, there are manifold uncertainties in the later implementation, and particularly the instantiation of the ontologies and the reasoning system likewise. While the process model, including its process ontology and the central capability profile (compare Section 3.2) are already defined, relations between the capabilities (see Section 4.1), and the transitions and states in OCEO (see Section 4.3) are yet to be fully defined. The major challenges in modeling the proposed system are in the design of the hidden Markov layers. By Bayesian optimization, it is possible to define the probabilistic dependencies between the emissions and the hidden states, i.e., the capability qualifications. However, emissions have to be modeled manually, which is more complicated than in more modern learning methods, in which features and intermediate states are connected with minimal supervision. On the contrary, in HRC, it is desired to have more explainable AI [24,25], particularly when it comes to direct interaction. Therefore, we expect the HMM to yield more explainability and acceptance in later deployment. However, it remains to be seen how well the emissions can be defined, and how large the uncertainties will be in the final implementation. Note that we assume that there will be fewer features than the total number of qualifications of capabilities, and hence, the instance of the OCEO will be under-determined. To this end, of particular importance will be the definition of the relations within the capabilities (e.g., see Figure 4). Well-defined dependencies will reduce the uncertainties propagated from the HMM and the number of degrees of freedom of the whole OCEO instance. To this end, we will also reduce the number of capabilities m defined in the capability profile, which again will be performed after a profound modeling of dependencies to assess the influence of the capabilities on the matchmaking decision.

Besides the technical aspects, there are also social and ethical uncertainties. While we assume that such a matchmaking system will raise the efficiency and acceptance of HRC workplaces in manual labor tasks, the opposite may manifest. In particular, when able-bodied workers and PwD share the same work place, and the situation occurs in which the system reallocates certain tasks from the able-bodied worker to the robot (e.g., caused by exhaustion or distraction), the interacting worker may become confused or skeptical. This behavior is expected in the first interactions. However, we assume that over a longer period of working with the HRC workplace, a learning process will take place that lets the worker appreciate the technology and its supportive actions rather than being repelled by it. We further assume that by establishing a work process capable of supporting able-bodied workers and PwD likewise, also the acceptance toward PwD will be increased. This is what makes full inclusion in first labor market processes possible in the first place.

In the following years, we will delve further into the methodology. We aim to implement and evaluate, in particular, the matchmaking ontology RAMO. Therefore, we hope to facilitate a discussion on the topic and to use this paper as a vehicle for the improved inclusion of PwD in HRC processes of the first labor market.

7. Conclusions

In this work, we first introduced how capabilities of PwD are modeled and how their dynamics evolve over the course of a workday and over a long time span. To this end, we introduced two time concepts $\tau$ and $t_0$ which eventually manifest in a Langevin system. The emerging capability profile $\mathbf{p}$ states the basis to a two-sided matchmaking ontology consisting of the two sub-ontologies: the process ontology and the observable capability emissions ontology (OCEO). While the process ontology models the qualification of sub-sets of capabilities, so-called basic elements $b_k\subset \mathbf{p}$ from the work task T, OCEO models how the full capability profile is derived from features $g_j$, originating from sensors, meta data, and system states. Based on the combined ontology, robotic assistance matchmaking ontology (RAMO), people may be observed and matched with process requirements to assess the specific in situ need of assistance. In addition, we showed how the ontology is used in automated task planning to generate dynamic work schedules based on the ontology-based reasoning approach. This results in a system description that allows dynamic and intuitive HRC not only for PwD, but for everyone.