Development of an Autonomous Robot Replenishment System for Convenience Stores

Abstract

With the appealing ability of combining both mobility and manipulability, mobile robot manipulators have been applied to factory automation, telemedicine, warehousing, etc. Due to the high density of convenience stores in Taiwan, Asia, and other countries, we make further use of mobile robot manipulators to develop an autonomous replenishment system. The replenishment task for a convenience store poses challenges given its narrow aisles and the occasional presence of customers. It thus demands thorough consideration of the planning, navigation, and control of the robot, in addition to customer safety. Correspondingly, we propose strategies for task and path planning, in addition to control and sensor fusion. Specifically, a new collision-free path planning algorithm (the DWA-PS) that consider both customer safety and comfort is developed, along with replenishment and control strategies that utilize the flexibility of the mobile robot manipulator to raise working efficiency. For the performance evaluation, we apply the proposed system to conduct a series of replenishment experiments, including in a convenience store near our university.

1. Introduction

Mobile robot manipulators have gained popularity owing to their salient combination of both the mobility of a moving platform and manipulability of a multijoint robot manipulator. They thus fit well for applications such as retail business, warehousing industry, medicine and meal delivery in hospitals, etc. In this paper, we further exploit their capability in order to develop an autonomous replenishment system for a convenience store. Facing worldwide shortages of labor and the spread of pandemics, the success of such an automated system should also be beneficial for the general retail business [1].

Convenience store management is labor-demanding, and Taiwan ranks second in density of convenience stores in the world, with an average of one for every two thousand people. Considering feasibility, robots are better suited for tasks such as inventory checking, replenishment, customer interaction, toilet cleaning, etc. [2]. Among them, automatic replenishment is our focus, as it is effective in resolving the out-of-stock situation, which is closely related to the operation of a convenience store. However, the replenishment task is very challenging, because the aisles between the shelves are in general very narrow and customers may be in the vicinity.

In previous related studies, Zhang et al. proposed using CNN to train a supermarket shopping robot for product recognition [3]. Gross et al. proposed designing an interactive shopping robot for customer guidance [4]. Additionally, Yedla et al. proposed a learning object detection system [5]. In practical business applications, FamilyMart in Japan has installed a replenishment robot system that allows the clerk to replenish goods via VR remote control [6]. Meanwhile, Decathlon in San Francisco has stationed shelf-scanning robots that can track the inventory status in real time [7].

Even with this progress, an autonomous robot replenishment system for a convenience store is still in demand, especially effective strategies for robot path planning, navigation, and control, along with careful handling of customer safety. To date, there has not been much research on collision-free path planning that is directly devoted to convenience store or supermarket environments. Lewandowski et al. proposed a socially aware robot navigation system for supermarket environments with the DWA algorithm and personal space adopted for path planning [8]. Their strategy is relatively conservative, as it would keep the robot distant from the customer during task execution. For the implementation of smart supermarkets, Zhang et al. proposed a hybrid solution based on actuator networks and IoT [9]. The robots in their design serve as the shopping assistants, so that the guidance system is more concerned with how to respond to service demand rather than possible collisions with customers. To deal with small working spaces and various goods in a self-service supermarket, Hu et al. proposed a trajectory planning method in the joint space of a six-DOF goods-picking robot manipulator to realize automatic goods picking [10]. However, their main consideration is to ensure the robot arm does not collide with obstacles, rather than a person.

In this paper, we first come up with a task planner for determining obstacles’ corresponding order on and off the shelf. By utilizing both the movements of the mobile platform and robot manipulator, we also propose a new collision-free path planning algorithm based on the dynamic window approach (DWA) [11], named DWA-PS, along with a control scheme based on model predictive control so that the robot can not only be able to avoid collisions with customers but also maintain its working efficiency. To demonstrate its effectiveness, we apply the proposed system for the replenishment task in a convenience store near our university. Compared with previous approaches, we consider the proposed system to have the following merits:

• A new collision-free path planning algorithm, DWA-PS, is developed that considers both customer safety and comfort.
• Novel replenishment and control strategies are developed for coordinating the movement of the entire mobile robot manipulator to achieve higher working efficiency under the strict constraint of customer safety.
• Via integration with the developed task planner that can determine a proper order for replenishment and the sensor-fusion method that leads to the required accuracy in both position and orientation, the proposed system is capable of conducting a field study at a convenience store.

The rest of this paper is organized as follows: Section 2 describes the proposed system, including modules of task and path planning, control scheme, and multisensor fusion. Experimental results in our laboratory and also a convenience store are reported in Section 3. Finally, concluding remarks are given in Section 4.

2. Proposed System

Figure 1 shows a conceptual diagram of the proposed autonomous replenishment system based on the mobile robot manipulator, which includes modules for task plans, navigation and control, and sensor fusion. For its daily operation, the proposed system will routinely send out the mobile robot manipulator to check the shelves for goods that are out of stock and then determine a proper order for replenishment via the task planner, described in Section 2.1. A corresponding path will then be planned, which consists of two parts: one for reaching a given shelf and the other for replenishment. Both of them need to deal with the presence of the customers, but they have their respective constraints during navigation and replenishment, and thus solicit different strategies, as described in Section 2.2. The path for navigation will be converted into motion commands for execution directly, while for replenishment, which involves movements of both the mobile platform and multijoint robot manipulator, it will be further modulated by the model predictive controller, described in Section 2.3. Meanwhile, the module for sensor fusion is utilized to achieve the desired accuracy for execution of the planned path, described in Section 2.4.

2.1. Task Planning

The goal of the task planner is to find a replenishment order that leads to the effective supply of out-of-stock goods. For this purpose, the system adopts the A* algorithm to look for the shortest path among all possible routes. First, the robot conducts a daily patrol of the shelves, and the system records the types and quantities of missing goods, along with their corresponding locations on the shelves, and stores them in a database. Assuming that there are N items out-of-stock goods, and the robot can carry m goods in one run, the A* algorithm will then need to search for the shortest path among N permutations of goods, with m goods for a trip. As the number of searches may be very large, two rules are set up in advance to remove unnecessary searches: (1) goods of the same type and (2) goods on the same shelf must be adjacent, as illustrated in Figure 2. Finally, the combination that corresponds to the shortest path will be used as the goal for path planning by utilizing the strategies described in Section 2.2. Based on the aforementioned plan, the algorithm for effective replenishment is organized as follows:

Step 1:

Send out the robot for patrol and record the information of the missing goods into a database.

Step 2:

Among all N permutations of the goods, retain those conforming to the two rules.

Step 3:

From the remaining combinations, apply the A* algorithm to find the combination that results in the shortest path.

Step 4:

Output that combination as the goal for path planning.

2.2. Navigation and Replenishment

After the task planner comes up with the most efficient replenishment order, the system proceeds to plan paths in the environment of a convenience store. In responding to different constraints for navigation and replenishment, two kinds of planning strategies are proposed accordingly. For navigation, the presence of customers and staff is mainly taken into consideration. The proposed strategy should not only prevent the robot from colliding with them but also keep a distance to alleviate feelings of oppression, as described in Section 2.2.1. As for replenishment, in addition to the safety concern, the strategy will also take advantage of the mobility of the mobile robot manipulator to maintain its working efficiency, even when people are close by, as described in Section 2.2.2.

2.2.1. Navigation

With the order of replenishment derived above, the system moves on to plan a path that can reach the target shelves sequentially and dynamically adjusts its speed to avoid collisions with customers during navigation. Among previous research on mobile robot navigation, Everett, Chen, and How proposed applying deep reinforcement learning for collision avoidance among a variety of types of dynamic agents without assuming they follow any particular behavior rules [12]. Li et al. also proposed an online learning navigation method based on deep reinforcement learning for obstacle avoidance in an unknown environment [13]. Liao et al. proposed the Stack-RRT* and combined it with different parameter curve-based smoothing schemes for smooth path planning [14]. For those related to human-aware navigation, Ferrer, G. and Sanfeliu proposed applying the extended social force model for robot navigation in crowded urban environments [15]. Weinrich et al. proposed conducting lifelong learning of people’s behavior for the mobile robot to adapt its behavior [16]. Chen et al. proposed applying deep reinforcement learning to model human–robot and human–human interactions occurring in dense crowds [17]. Additionally, Kruse et al. conducted a survey of the approaches for human-aware navigation [18]. While these studies were intended for mobile robots, the concepts are helpful in designing the proposed safety and comfort schemes based on mobile robot manipulators.

We first utilize the A* algorithm to plan a path to reach the target shelf in the presence of static obstacles, e.g., the shelves along the aisle or walls, and then assign the mobile platform with its maximum allowable speed for path following. However, when a customer comes up, its speed needs to be adjusted to ensure not only collision avoidance but also customer comfort. To meet these two demands, we propose a novel algorithm based on the dynamic window approach [11], the DWA-PS, which also takes into account the effect of the personal space (PS) of the customer [19], discussed later.

Figure 3 shows how the A* algorithm and DWA-PS are combined together to derive a path with an appropriate execution speed. In Figure 3, the red line indicates the path S^A planned by the A* algorithm that serves for the DWA-PS to determine proper linear and angular velocity $V_B$ ($v_B$, ${\omega }_B$), which may lead to a new path S^B (the green line) that deviates from the original one due to the incoming customer for collision avoidance. With the same S^B path to follow, the DWA-PS will further slow down or maintain the speed $V_C$ in response to customer comfort. Details of the A* algorithm for path planning are omitted here because of its popularity, and those of DWA-PS are as follows.

As a famous online collision avoidance strategy for mobile robots, the DWA utilizes velocity for path planning. To determine the linear and angular velocities ($v$, $\omega$) along the path, three main factors need to be considered: the angle ($ang$) between S^A and S^B, indicating their closeness, the distance ($dis$) between S^B and the nearest customer, and the execution velocity ($vel$). The goal is to find the maximum allowable ($v$, $\omega$) that may minimize $ang$ but maximize $dis$. As customers may show up unexpectedly, the DWA-PS is set to adjust speed for every m seconds, and it will run for n times during that period (m = 2 and n = 20 were used in the experiment). Among the n simulated paths, S^B is chosen to be the one with the highest score according to cost function $G\left(v,\omega \right)$:

$$G\left(v,\omega \right)=\alpha ·\left(\pi -ang\left(v,\omega \right)\right){|}_n+\beta ·dis\left(v,\omega \right){|}_n+\gamma ·vel\left(v,\omega \right){|}_n$$

(1)

where ($\pi$-ang($v$,$\omega$)), dis($v$,$\omega$), and vel($v$,$\omega$) (both $v$ and $\omega$ included) should be normalized for the n simulations to be compared on the same basis, symbol “${|}_n$” stands for normalization, and $\alpha ,\beta ,\mathrm{and}\gamma$ are the weighting factors to be determined according to the designer. Note that dis($v$,$\omega$) should be at least 0.8 m for safety concerns, and both $v$ and $\omega$ are limited to allow enough time for deceleration to prevent the mobile platform from hitting the customer.

For $V_B$ that corresponds to S^B derived above, it will be further modified to be $V_C$ based on the concept of personal space, which is defined as a physical area surrounding an individual that is considered personal or private. People usually do not like others, including robots, to stay too close them. According to [19], it is considered to be uncomfortable when the relative distance D between a person and robot is within 0.46 m and to be all right when beyond 3.7 m, while feelings vary between individuals. Consequently, we set $V_C$ = $V_B$ when D > 3.7 m and $V_C$ = 0, i.e., a full stop when D < 0.46 m. In between, the speed will decrease gradually along with the closeness of the person to the robot.

Figure 4 shows an illustration of the personal space, which has a larger front area, formulated as half of an ellipse, as people care more about the incoming object. Its influence on speed adjustment depends on the relative distance D and orientation $\theta$ between the customer and robot. We use a Gaussian distribution function $f\left(x,y\right)$ for its formulation, where $\left(x,y\right)$ is the position of the robot, with $\left({\sigma }_x,{\sigma }_y\right)$ as the standard deviation along the x and y axis, respectively, selected as 2.8 and 0.24 to match the values of 3.7 and 0.46 for D, and formulate it as

$$f\left(x, y\right)=e^{-\left(f_x+f_y\right)}$$

(2)

where

$$f_x=\frac{{\left(D\ast cos \theta \right)}^2}{2\ast {\sigma }_x{}^2}, f_y=\frac{{\left(D\ast sin\theta \right)}^2}{2\ast {\sigma }_y{}^2}$$

(3)

From (2) and (3), $f\left(x,y\right)$ reaches a maximum value of 1 when $\left(x,y\right)$ = (0,0) and decreases as D and $\theta$ increase. We utilize this property to come up with the weighting factor $ps\left(x,y\right)$ for $V_C$ adjustment, described in (4) and (5):

$$V_C=ps\left(x, y\right)\ast V_{B_{ }}$$

(4)

with

$$ps\left(x, y\right)=\left\{\begin{array}{cc}\hfill 0,& D\le 0.46\hfill \\ \hfill 1-f\left(x, y\right),& 0.46<D\le 3.7\hfill \\ \hfill 1,& D>3.7\hfill \end{array}\right.$$

(5)

2.2.2. Replenishment

After the robot reaches the target shelf via path S^B derived above, the system continues to plan the path for replenishment, which considers both customer’s safety and working efficiency. According to the distance between the robot and customer, we specify several safety zones around the robot [20], as shown in Figure 5. When a customer is present inside the stop zone, the robot needs to be at full stop and decrease its speed within the deceleration zone, while the speed is maintained when staying in the normal zone. The size of the stop zone, subject to variation, is set to be 1.5 m, which is determined based on the speeds of the mobile platform and the customer (set as 1.6 m/s), as well as the working range of the robot arm. By adding another 0.5 m, it forms the deceleration zone, which serves as a buffer between the normal and stop zone. To allow a smoother transition from the initial speed $V_{in{i}_{}}$ in the normal zone to that in the deceleration zone $V_{dec}$, it is determined by following a sigmoid function, described in (6) and (7):

$$V_{dec}=de\left(D\right)\ast V_{in{i}_{ }}$$

(6)

with

$$de\left(D\right)=\frac{1}{1+e^{-\beta \left(D-1.75\right)}}$$

(7)

where $de\left(D\right)$ is the weighting factor, with D as the distance between the customer and robot and $\beta$ as a constant, which is selected to be large enough to make $de\left(D\right)$ approximate 1 when D > 2 m and 0 when D < 1.5 m, as shown in Figure 6.

To raise the working efficiency when a customer is present in the deceleration zone, the proposed strategy is to move the mobile platform along the opposite direction a certain distance, so that he/she would then be located at the normal zone to have a higher execution speed. As the aisle is in general very narrow and the robot arm also needs to be close to the shelf, only horizontal movement is taken, as shown in Figure 7. In Figure 7, with the mobile platform moved to leave a distance of d to change that between the customer and robot from $D_{1_{}}$to $D_2$, the customer is now located at the normal zone (outside of the red line), instead of the original deceleration zone (between the green lines). Note that the length of d is limited to be about half of the moving range of the robot arm in the horizontal direction, as the robot arm still needs to have enough space to conduct the replenishment task.

2.3. Model Predictive Control

The path for navigation, along with its speed assignment derived above, involves only the movement of the mobile platform and can be converted into corresponding motion commands sent to the robot controller directly. However, that for replenishment would solicit simultaneous movements of both the mobile platform and multijoint robot manipulator and needs to be further modulated before its execution. We thus propose a controller based on model predictive control (MPC) for their coordination, which is deemed to be effective in dealing with multivariable and multirestricted planning problems [21].

Figure 8 shows the system block diagram for the proposed MPC-based controller, which aims to derive the optimal control input that leads to the goal under system constraints and cost function. In Figure 8, the module of path planning first designs two paths, one for the mobile platform to move distance d intended for raising working efficiency, described in Section 2.2.2 above, and the other for the robot manipulator to conduct replenishment by utilizing cubic polynomial for achieving velocity continuity, denoted as $X_D$ as a whole. The optimizer will then derive the optimal control command U sent to the mobile robot manipulator according to system constraints and cost function. Finally, robot state X, along with U, will be sent to the predictive model to predict a future path $X_p$. The process will be executed repeatedly until path $X_D$ is finished.

The constraints are set to be the maximum and minimum of the working range of the robot manipulator, as well as those for the velocities and accelerations for both the robot manipulator and mobile platform. The cost function J for the optimizer is intended to minimize the errors between the desired and the predicted position and velocity along the path, chosen to be

$$J={\displaystyle \sum _{i=0}^N}\Delta X^T{W}_X\Delta X+{\displaystyle \sum _{i=0}^N}\Delta U^T{W}_U\Delta U$$

(8)

where$X={\left[\theta P\right]}^T$ and $U={\left[\omega V\right]}^T$ with $\theta =\left[{\theta }_1\dots {\theta }_6\right]$ and $\omega =\left[{\omega }_1\dots {\omega }_6\right]$, and$P=\left[p_x{p}_y{p}_z\right]$ and $V=\left[v_x{v}_y{v}_z\right]$ stand for the joint angle and velocity of the robot manipulator and position and velocity of the mobile platform, respectively, $\mathsf{\Delta }X=X_D\left(k+i\right)-X\left(k+i\right)$ and $\mathsf{\Delta }U=U_D\left(k+i\right)-U\left(k+i\right)$ with k indexing a given moment, $N=T_p/T_s$ with $T_p$ as the period for prediction and $T_s$ the sampling time, and $W_x$ and $W_u$ are the weighting matrices. The predictive model is formulated as

$$X\left(k+1\right)=AX\left(k\right)+BU\left(k\right)$$

(9)

where A is selected to be $I_9$, a unit matrix, and $B=t_s\ast I_9$ with $t_s$ as the sampling time. Finally, the process for finding an optimal U is formulated as a quadratic programming problem and solved by utilizing the OSQP in [22] in real time.

2.4. Sensor Fusion

To achieve the required accuracy in both position and orientation for path navigation, we equipped the LiDAR, inertial measurement unit (IMU), and odometer sensors on the mobile robot manipulator. The LiDAR is considered to have positional advantage and is also fast in data collection, and its combination with IMU and odometer can lead to higher position and orientation accuracy, especially the latter. Due to their characteristics, e.g., the IMU is susceptible to drifting from environmental interference and the odometer is prone to accumulated errors, their effectiveness on measurement varies under different conditions. Therefore, we propose a method for effective sensor fusion.

During the measurement process, the odometers equipped on the wheel are used to provide the initial position and orientation and the LiDAR and IMU for later updating along the navigation. To determine which sensing data should be used at each sampling time, we compare the moving distance and angle of the robot $p_M$($\Delta d_M$, $\Delta {\vartheta }_M$) measured by the sensors with $p_P$($\Delta d_P$, $\Delta {\vartheta }_P$) predicted by the extended Kalman filter (EKF) [23]. The comparison is based on the Mahalanobis distance (MS) [24], which is basically a normalized Euclidean distance in multidimensional space, and a small (large) MS implies a better (worse) prediction from the EKF. Consequently, we set the rule for measurement data selection as

When the MS for the position (orientation) is smaller (larger) than the chosen threshold value for position (orientation), the measurement from the LiDAR (IMU) is used for the next prediction, otherwise, it will be ignored.

By applying the proposed sensor fusion method and the adaptive Monte Carlo localization algorithm [25], which is efficient for adjusting the sampling points to adapt to environmental changes, the mobile platform achieves an accuracy of less than 4 cm and 1 cm in the direction along the aisle and that facing the shelf, respectively, which is much better than 10 cm by using the LiDAR alone for mobile robot navigation, as reported in [26]. Meanwhile, it reaches an accuracy of less than $3^{\circ }$ in orientation.

3. Experiments

We conducted a series of experiments to evaluate the performance of the proposed system, including a field study in a convenience store. In order to recognize the customers and objects for replenishment during the experiments, the system first analyzed point cloud information collected by the equipped depth camera [27], with examples of customer and bottle images shown in Figure 9, followed by coordinate transformation to obtain their relative position with the robot. In the study, we only dealt with bottle-type objects, which are solid and easy to grab. In our next stage of research, this will be extended to objects of various shapes and stiffnesses. To support real-time execution, the proposed system is equipped with a computer with an Intel Core i5-6500TE CPU, an Nvidia GeForce GTX 1650/4GB, and the Ubuntu 16.04 operating system, along with the ROS Kinetic.

As there may be plenty of items to be replenished in a convenience store, we first examined the effect of the proposed task planner described in Section 2.1. Even with only a few items, the number of possible replenishment orders can be very large. Taking 6 missing items for 4 shelves as an example, and assuming a case of 3, 1, 0, and 2 missing item(s) on each of them, it can lead to 360 combinations in total. By applying the algorithm for effective replenishment, the combinations can be cut down to 24 feasible ones, which allows the A* algorithm to locate the desired replenishment order with the shortest path more efficiently.

We started with the first set of experiments to check whether the proposed system could guide the robot to reach the target shelves with the desired accuracy. Figure 10a shows a scene in our laboratory that emulates the environment in a convenience store, and Figure 10b the corresponding simulated one. It was arranged to have one warehouse (number 0 in blue) and four shelves (number 1–4 in orange), with one missing item for each shelf. In the path shown in Figure 10b, the robot started from the warehouse, conducted replenishment from shelf 1 to 4 sequentially, and then returned to the warehouse. A total of five laps were executed.

Table 1. Position error in X direction (unit: cm).

Lap	Shelf 0	Shelf 1	Shelf 2	Shelf 3	Shelf 4	Average	Deviation
1	0.57	−0.57	0.89	−3.60	2.09	−0.12	2.16
2	−0.31	1.38	−2.72	−0.48	−1.26	−0.68	1.49
3	−0.32	0.98	−0.19	−1.96	−2.03	−0.71	1.28
4	0.41	−1.18	−2.21	−0.86	−2.71	−1.31	1.22
5	0.26	2.59	3.51	−1.91	1.53	1.19	2.12

and

Table 2. Position error in Y direction (unit: cm).

Lap	Shelf 0	Shelf 1	Shelf 2	Shelf 3	Shelf 4	Average	Deviation
1	0.07	−0.08	0.33	−0.08	−0.08	0.03	0.18
2	−0.17	0.13	−0.14	0.14	−0.09	−0.02	0.15
3	−0.14	−0.02	−0.43	0.09	−0.31	−0.16	0.21
4	0.01	−0.12	−0.31	0.20	0.15	−0.01	0.21
5	−0.03	0.09	−0.54	−0.04	0.03	−0.10	0.25

list the position errors in reaching the shelves in X and Y direction to be less than 4 cm and 1 cm, respectively, where X is the moving direction and Y that for replenishment. As the accuracy achieved satisfied task requirements, the system was ready for the following experiments in the real environment.

The second set of experiments were conducted in a convenience store near our university to investigate how the proposed system would respond when the staff or customers showed up during replenishment. Figure 11a shows a situation in which the robot needed to pass a customer on the way to the target shelf. To avoid collision with the customer and also consider his personal space, the system activated the DWA-PS algorithm described in Section 2.2.1, which would deviate the robot from the original path and also slow down its speed. Figure 12 shows the corresponding speed adjustment, which occurred between 4.5–9.5 s. After that, the robot returned to its normal speed and continued following the planned path, as shown in Figure 11b. Figure 13a shows another situation in which the robot faced an incoming customer and needed to pass him. Figure 13b shows the resultant paths by applying the DWA and DWA-PS algorithm, respectively. With the person’s space taken into account, the DWS-PS led to a path that kept a larger distance when passing the customer when compared with that of DWS. For the case of a customer coming up from behind, the robot would maintain the original speed when he/she moved slowly and slow down and slightly turn to the side for safety when the customer moved fast to pass by.

To evaluate the performance of the proposed replenishment and control strategies, Figure 14a shows a situation where a customer approached from the other side of the aisle when the robot was conducting replenishment. As the safety zones were set up around the robot, it needed to slow down and even stop when the customer stepped into the deceleration or stop zone. To raise working efficiency, the system then moved the mobile platform forward to let the customer be located within the normal zone, as shown in Figure 14b, so that the robot manipulator could still maintain the same speed to replenish. Figure 15a shows the relative distance between the customer and the robot during the process, where the blue and red lines indicate it before and after the mobile platform movement, respectively. In Figure 15a, the customer entered the deceleration zone at 7 s, which solicited the mobile platform to move 20 cm forward, as shown in Figure 15b, and later at 17 s, he further stepped in another 10 cm, which again propelled the mobile platform to move forward the same distance, letting the robot still be able to conduct the replenishment task.

For further investigation, we conducted the third set of experiments for cases involving multiple customers. Figure 16a shows the situation in which the robot needed to pass three customers during navigation. The proposed system successfully found a collision-free path, as shown in Figure 16b. However, when the aisle was too crowded for the DWS-PS to derive a feasible path, it might search for an alternative one first if possible; otherwise, the proposed system would simply command the robot to stop and wait. Figure 17 shows a scene in which the robot managed to conduct replenishment between two customers. Both of them entered the robot’s deceleration zone and stopped at positions approximately 1.6 m and 1.9 m from its front and back side, respectively. Because the target shelf was located close to the customer on the left, the proposed system moved the robot to the right to make room for raising the replenishment speed. Meanwhile, to ensure the safety of the customer on the right, the robot was constrained not to move too close to him. Figure 18 shows the process of distance adjustment for the robot to reach a proper location between the two customers. In sum, successful execution of these three sets of experiments demonstrates that the proposed system was able to execute the replenishment tasks in the presence of customers under the demands of both safety and working efficiency. Please find the connection to the video that shows the experiments conducted in the convenience store at https://youtu.be/xXabdGyErKs in Supplementary Materials.

4. Conclusions

In this paper, we proposed an autonomous replenishment system based on a mobile robot manipulator and applied it to a convenience store. To achieve safe navigation in narrow aisles and maintain working efficiency in the presence of customers, we specifically developed the DWA-PS path planning algorithm that considers both customer safety and comfort and the replenishment and control strategies that well utilize the flexibility of the mobile robot manipulator to raise working efficiency, along with a task planner and a sensor fusion scheme for system integration. Extensive experiments involving single and multiple customers under various situations were conducted for demonstration, including ones which were actually executed in a convenience store. For further study, we plan to enhance the capability to deal with more complicated and challenging environments.