Autonomous Movement of Wheelchair by Cameras and YOLOv7 ^†

Abstract

A wheelchair can provide limited but crucial mobility to an injured or disabled individual. This paper presents the first stage of the development of a smart wheelchair which is the customization of a manually controlled wheelchair with a novel implementation of octascopic vision. This relatively inexpensive design of an autonomous wheelchair consists of two monochromic camera arrays (each having four cameras) placed around the frame of the wheelchair to achieve a view of 360 degrees. The initial research goal was to design a wheelchair controlled by the embedded processor, allowing the wheelchair to navigate autonomously around an indoor facility with and without human intervention. Additionally, it was intended to allow those previously denied access to the world of automatic wheelchairs because of a low personal income. Through the testing of wheelchair functionality, (a) a large dataset of octascopic images was captured from this wheelchair, and (b) a YOLOv7-based object detection model was developed to avoid obstacles and autonomously control the movement. This paper presents the camera placement and the obstacle detection model using octascopic images. All the project design files have been granted an open-source license and can be reproduced publicly.

1. Introduction

According to the World Health Organization, 15% of the world’s population lives with some form of physical disability [1]. Currently, 61 million adults in the US live with a disability [2]. It means one out of four adults in the US has disability issues. This muscular illness, injury, or disability may cause depression, significant decreases in motivation, and loss of independence for many sufferers. Responding to this, a few mobility-assisting devices have become available in the market, such as wheelchairs. Hence, our primary research motivation was to give those people their autonomy back, allowing them to move safely when and where they want without requiring assistance from caregivers or others.

Studies have shown that cameras are widely used in cars for parking, lane detection, autonomous movement, vision-enabled prosthetic hand, etc. [3,4,5,6,7,8,9,10,11,12]. A combination of LiDAR and cameras also used in navigation of wheelchair [13,14,15,16,17,18,19]. Additionally, it is also noticeable that the Convolution neural network (CNN) is also popular among researchers [20,21,22,23]. YOLOv7 gained popularity in object detection among researchers. The benefits of YOLOv7 include quick convergence, excellent precision, and robust customization. It may also be easily transferred to embedded devices due to its powerful real-time processing capabilities, and low hardware computing needs [24].

Although many projects have been found for autonomous navigation for cars in the outdoor environment, there is less study on eight camera-based autonomous wheelchairs for indoor environments. Because of the structure of the wheelchair, it is harder to place a LiDAR-based system to cover ~360 degrees. Our target was to work in this area to build a robust solution for the autonomous wheelchair using a vision-based solution with multiple cameras and a deep neural network.

The contribution of this study is the presentation of a vision sensor (eight OmniVision camera combination) integrated smart wheelchair which is capable of detecting obstacles from the captured octascopic images. This is completed by implementing the YOLOv7, a deep learning-based object detection model. The project’s design files have been made available in [25]. The design concept is that the eight cameras cover ~360 degrees view and while moving the object detection model will detect any object and send the appropriate commands to the motor controller to move the wheels accordingly.

This paper is organized as follows: Section 2 describes camera placement detail. Description of YOLOv7 deep learning model training and accuracy in Section 3. The discussion is in Section 4, and the conclusion is in Section 5.

2. Camera Installation

The next step was to install eight 1-megapixel monochromic cameras, each having a view angle of 75 degrees. Six cameras were placed in the front, with three on each side, and two cameras were placed in the back for rear view. The cameras were placed in such a way that they were close to achieving ~360 degrees of view. In each array of three cameras placed in the front, one camera faces back at a 15-degree offset from the wheelchair. At the rear of the wheelchair, two cameras were placed 180 degrees from the cameras viewing the front of the wheelchair. Figure 1(left) shows the top view and coverage of the cameras. There were some overlaps, but we tried to reduce blind spots.

There was no camera mounting point in the original wheelchair. Custom parts were made to set the cameras into the wheelchair. Because the Jetson Nano only has two digital camera interface MIPI ports, the eight cameras were connected through two Arducam quadrascopic monochrome camera arrays. Figure 1(right) shows a set of Arducam quadrascopic camera arrays. The right three cameras and right-back camera are connected to one array, and the left three and left-back cameras are connected to another Arducam camera array. Then two Arducams quadrascopic arrays were connected to the Jetson Nano MIPI camera ports. Each camera array needs a 5V DC supply connected to the buck converter output.

3. Obstacle Detection Model

Since the study goal was real-time object detection, we employed YOLOv7 [26], the most recent release of the single-stage object detector YOLO (You Only Look Once [27]) models. As they are small and trainable on a single GPU, YOLO models are better for modeling object detection. Comparing its cohorts, YOLOv7 provides faster (>5 FPS on a V100 GPU) and greater accuracy in predicting the bounding box of the objects (obstacles in our study), thus taking state-of-the-art to new heights. The detailed architecture is provided in [26]. The YOLOv7 network is defined in PyTorch, and training scripts, data loaders, and utility scripts are written in Python. The original model was trained to detect the generic 80 classes in the COCO dataset. For training, we have used custom classes (human, chair, table, pole stairs, trash can, door, notice board with stand, cart, dead end, and wall) and annotated using open-source library Labelimg [28]. Figure 2 shows an example of the annotated image that we collected using our wheelchair.

In our study, we re-trained the YOLOv7 model for 10 object classes. The images were resized to 1280 × 400 before training. While training, a batch size of 16, 100 epochs, and adam optimizer were chosen with multi-scale and hyperparameter evaluation enabled. Figure 3 shows the trained model’s output that it detected surrounding objects.

We have trained the model with our custom annotated dataset for 200 epochs and obtained 92 mAP. Figure 4 shows the confusion matrix of the model. where we can see the model successfully detected objects in most cases. There is some false detection that can be seen from the matrix those are due to less annotated data. We believe adding more annotated data can increase the model’s accuracy.

4. Discussion

This study presents a novel application of vision sensor arrays to obtain an octascopic view of the environment and detect obstacles for autonomous movement; this could be a promising future for wheelchair technology. The design features simplicity and flexibility, and it depends on the commercially available off-the-shelf components. Open-source components and software may enable the community to push this design further as new techniques and approaches become available. Additionally, the end users can install the components to their wheelchairs and convert them to autonomous ones. In our case, we bought the commercially available wheelchair for USD 150, and the total cost required to convert the commercially available wheelchair is USD 1300. In our previous study, we implemented electroencephalogram (EEG) [29] on a wheelchair, and to provide autonomous movement we have implemented the camera-based solution.

Here, we have used eight cameras to obtain ~360 degrees views that will help to detect if any object comes from any direction. If there is no way to go to the front side, then the system can determine the path by analyzing another camera. Another benefit of having multiple cameras is that if one camera is damaged or blocked, the other cameras that cover that area can be used as a substitution as we plan to run all the processing into embedded computers. We have used 1MP cameras.

More work on efficient model development for object detection needs to be completed to ensure low power and robust operation. The object detection model that will run the wheelchair needs to be perfected. This study was limited to obstacle detection, an ROS-based wheelchair operating system is required to introduce faster movement and automatic course correction. As a Jetson-based system is implemented and the battery may run out soon, a wireless charging system is required to be installed.

A large human study is required to test the wheelchair with disabled individuals. In addition to object avoidance, some other implementations can be performed, such as voice commands, brain-computer control, etc., to support the user with low or no mobility of arms to reach the desired destination. Innovations like this mitigate the gap between technology and humanity, arguably the purpose of technology.

5. Conclusions

The design of an autonomous wheelchair implementing octoscopic vision has great potential as it is capable of achieving ~360 degrees of view from its surroundings, other technologies, such as LiDAR, have flaws in that placement of LiDAR in the wheelchair is harder, and they can easily be blocked by the parts of wheelchair. By implementing an octoscopic array of cameras covering some partial view of the wheelchair should not considerably affect its performance as it still has the feed from the rest of the cameras. To increase the accuracy, we need to add more images to train the model. A ROS-based wheelchair operating system is needed to enable faster movement and automatic course correction because this study was restricted to obstacle detection. We will study Wi-Fi-RTT on the wheelchair for localization. Our modified wheelchair is cheaper than the cost of a standard electronic wheelchair. As the project design files are open source, anyone can download and implement them.