WARM: Wearable AR and Tablet-Based Assistant Systems for Bus Maintenance

Featured Application

Comparison two digital solutions (tablet based and Augmented Reality based) for bus maintenance against the traditional solution based on paper.

Abstract

This paper shows two developed digital systems as an example of intelligent garage and maintenance that targets the applicability of augmented reality and wearable devices technologies to the maintenance of bus fleets. Both solutions are designed to improve the maintenance process based on verification of tasks checklist. The main contribution of the paper focuses on the implementation of the prototypes in the company’s facilities in an operational environment with real users and address the difficulties inherent in the transfer of a technology to a real work environment, such as a mechanical workshop. The experiments have been conducted in real operation thanks to the involvement of the public transport operator DBUS, which operates public transport buses in the city of Donostia—San Sebastian (Spain). Two solutions have been developed and compared against the traditional process: one based on Tablet and another one based on Microsoft HoloLens. The results show objective metrics (Key Performance Indicators, KPI) as well as subjective metrics based on questionnaires comparing the two technological approaches against the traditional one based on manual work and paper.

1. Introduction

This work targets the applicability of wearable/mobile and augmented reality (AR) technologies for maintenance of bus fleets. These technologies have already shown its ability to enrich the human perception, increasing operators’ performance and reducing their error rate. In addition, the actual emergence of new augmented reality (AR) headset provides a better way of integration of such technologies for non-experts. Despite the advantages of AR, which have been proven in many previous publications, few companies have been quick to adopt this technology for industrial manufacturing applications due to the hardware for AR available in the market [1]. As the review shows, this area has only been sparsely covered by research. This might be due to the fact that devices powerful enough to support high-quality AR are only recently available at reasonable prices, and enabling technologies, such as ARCore and ARKit, have been released as free SDKs to developers only starting in 2017 [2].

Maintenance mechanics spent a valuable time reporting their activities, especially when unexpected reparations are accomplished. These reports have the form of handwritten annotations that have to be manually typed into the maintenance system by another maintenance worker. This manual data processing is error prone and often lacks completeness. In addition, only text information is reported; pictures, videos, and audio cannot be included.

Assistive systems could help workers to maintain or even increase their productivity [3]. These systems could be in the form of tablet-assisted systems or more sophisticated AR-based systems. Augmented reality is capable of projecting assistive instructions in-view of the user or directly in situ at the object of interest. The proposed assistant WARM helps the mechanics to report their activity, including rich multimedia data if so requested. The WARM assistant connects with the back office and with the vehicle through the IT standard EN13149 parts7/8/9 (https://www.en-standard.eu/pd-cen-ts-13149-7-2020-public-transport-road-vehicle-scheduling-and-control-systems-system-and-network-architecture/) for data exchange.

WARM has been demonstrated in DBUS garage, under real operational conditions. To this end, four maintenance actuations such as M1, M2, M3 (standard maintenance actuations) and PRE_ITV (prior to the official vehicle inspection) have been tested.

The main goal of the WARM assistant is, on one side, the reduction of the time spent by the maintenance workers in paperwork activities and, on the other side, the augmentation, enrichment and completion of the information reported by the maintenance operators at the end of the maintenance task. More in details, the system is committed to:

• Reduce maintenance reporting time. The maintenance activity requires not only for the “reparation task” but also for the “reporting task”. This paperwork usually comprises a checklist of 1–2 complete pages that has to be handwritten by the worker at the end of the task. In addition, the worker has to annotate relevant aspects of the task, particularly when an unexpected damage shows up and additional spares are requested.
• Allow maintenance operators to easily keep track and collect verbal comments, pictures, audio tracks and video clips concerning the particularities of the maintenance tasks accomplished, especially in the case of unexpected repairs. At present, the reporting consists of completing a checklist form.
• To demonstrate the feasibility of the standard EN13149 to exchange data between the maintenance assistance system and the vehicle.

In this paper, the authors have tested two different technologies for the WARM assistant, one based on tablet and another one based on AR headset, that make feasible the concept of “intelligent garage and maintenance”, targeting the applicability of AR technologies to the maintenance of bus fleets. The rest of the paper is structured as follows. The state of the art is presented in Section 2. The proposed solutions and their architectures are described in the Section 3. Section 4 and Section 5 show the description of the experiments and the results, respectively. Finally, Section 6 discusses and summarizes the main contributions of this work.

2. State of the Art

The main challenge of an AR system is to obtain perfect alignment between real and virtual objects in order to create the illusion that both worlds coexist. To that end, the position and orientation of the observer (i.e., the localization of the human with respect to the environment) has to be determined in order to configure a virtual camera that displays the virtual objects in their corresponding position. This problem is known as tracking and, although there are many alternatives to address it by using different sensors, tracking based on optical sensors is the most popular solution.

In the last years, markerless monocular solutions have gained in popularity due to their simplicity and low cost (one single camera) and by avoiding having to position markers in the scene. They take advantage of the visual cues that are naturally in the scene. Depending on whether the scene geometry is known or not, the markerless tracking is divided into two groups [4]. In Structure From Motion (SFM) approaches the camera movement is estimated while the 3D reconstruction of the scene is performed [5]. On the other hand, model-based techniques store the knowledge about the scene in a 3D model, which is available before the camera tracking begins. The 3D model could be represented by its simple 3D geometry [6], or by a more detailed description, that includes the geometry and the texture of its surface [7].

However, despite many years of research, optical tracking of industrial objects is not a solved problem (metallic objects with reflections, poor textures, dirty environment, etc.) and many studies have to be conducted yet [8,9] to provide reliable solutions that can be used in a factory. Errors in the estimation of the position of the virtual objects added to a scene greatly reduces the realism and quality of the integration of the information in the real context. This can severely limit the relevance of the information received by the user.

Focusing on industrial environments, AR solutions can also be found but, even markers-based solutions, most of them are in an experimental/lab prototype status [1,2,3]. Several works [8,10,11,12,13,14] demonstrate the benefits of AR based approaches applied to the assistance in maintenance, assembly and repair operations in terms of operation efficiency. These works argue that AR enriches the way in which users understand the real world, i.e., AR let users understand clearly what to do at any time. Compared to VR, AR offers information that is integrated into the real world, while VR manipulates virtual objects. For this reason, AR is used for both training as VR, but it is mainly used for guidance.

Some works [15] studied the long-term usage of high-quality AR technology in industrial training and determined that it reduces stress for the user compared to traditional training procedures, and therefore, improves worker satisfaction.

Current approaches use AR to present the worker with an “on-line” virtual manual of the task. In many cases, this is simply a list of the different steps that the user has to complete in order to finish a task. More advanced approaches present a virtual model integrated into the real world [16]. These works describe algorithms that recognize the 3D objects and track them meanwhile disassembly instructions are overlapping in the real scene guiding the user along the task. In this case, the disassembly sequence is computed in a pre-process following automatic methods based on the object geometry [17,18].

A very systematic review papers can be seen in [19,20,21] works. In all of these works, the common points are that AR has a lot of previous works showing tracking and registering applications in most of the market sectors. However, they agree that AR is still not mature for complying with industrial requirements of robustness and reliability. In fact, looking the experiments, we have the same conclusions in our work.

AR technology tries to improve the traditional concept of digital maintenance. Concerning to this, the maintenance management software market is highly fragmented with a poorly differentiated product, low-skilled support service, and very high availability. IBM has a comprehensive business asset management package for asset maintenance and lifecycle management called Maximo Asset Management [22]. The big drawback of this solution is its very high price. PRISMA 3 EAKM (Enterprise Asset Knowledge Management) [23] of Sisteplant is a 100% web application that provides intelligence to maintenance management and visibility aimed at all types of users from the perspective of life cycle management. MMS solutions (Maintenance Management Solutions) of Idasa [24] allow to quickly and efficiently manage equipment and facilities, assets and real estate, the available stock and its supply needs, maintenance services and service requests, own or contracted service personnel, as well as controlling the associated costs in detail and at all times. One of the largest provider of business applications and services in the world, Infor, has an EAM (Entreprise Asset Management Solutions) that improves maintenance schedules, increases manufacturing cell uptime, improves reliability and risk management policies, and provides deep insight within the company for more precise strategic planning [25]. Primavera’s solution allows planning, scheduling and managing maintenance according to the availability of technical means and the operational condition of the assets from anywhere with an Internet connection, thanks to its 100% Web platform [26]. As a summary, the following

Table 1. Feature comparison among different maintenance management software.

Name	Asset Management	Mobile Devices	Voice Recognition	Vehicle Identification	Augmented Reality
IBM	Yes	Yes	No	Yes	No
Sisteplant	Yes	Yes	No	No	No
Idasa	Yes	Yes	No	No	No
Infor	Yes	Yes	No	No	No
Primavera	Yes	Yes	No	No	No
WARM	Yes	Yes	Yes	Yes	Yes

shows the characteristics of the main systems and a comparison with WARM proposal.

3. Solution Approach

In this work, the use of wearable devices (tablet and AR device (e.g., smart headset)) is proposed. The devices do not need to be worn during the whole operation but just when required, to support the operator to complete the checklist and collect relevant multimedia data of the extraordinary facts of the activity.

The maintenance system currently deployed in DBUS is the system GIM (Gestión Integral del Mantenimiento, Integral Maintenance Management in English), which is a Computerized maintenance management system (CMMS) developed and commercialized by the company TCMAN. Recently, TCMAN has introduced the software GIM Android App to ease the collection of maintenance data by nomadic users. This application could benefit from some of the innovations pursued by WARM, as it lacks a friendly and efficient graphic user interface (GUI) well adapted to mechanic’s needs.

Figure 1 shows a schematic of the WARM system. Once the maintenance worker receives the work order sheet from the maintenance system (back office) using Web Services, the wearable device of the WARM system connects to the bus on-board computer through the WiFi network on-board. Using this connection, the wearable device identifies the bus and reads the relevant data for the maintenance task. This connection uses a proprietary interface. Additionally, to complete the data read from the bus, a direct connection to the CAN is done. At this point, the wearable device starts guiding the work to be completed, accordingly to the work order sheet and collects all relevant information. This guidance is adapted to the “work style” and preferences of each maintenance worker participating in the study. The worker will use the wearable device for different purposes:

• For dictating “done/undone” at the corresponding entries of the checklist form.
• For dictating annotations concerning the work done.
• For dictating the list of spares needed.
• For taking photographs and recording audio or video clips concerning the particularities of the reparation.

Once the work is finished and the report complete, this document is sent to the back office. All communications are compliant with the standard EN13149 via WiFi.

WARM system has been developed upon the application program interface (API) of the GIM system. Data collected with the WARM system are naturally uploaded and integrated in the DBUS maintenance servers running GIM. Besides, GIM API implements a communication layer to exchange information with GIM servers by means of Web Services. This layer has been adapted to comply with the standard EN13149. Finally, a new development was necessary to connect GIM Android APP with the bus (i.e., the CAN/FMS interface) using again the standard EN13149.

3.1. Tablet-Based Solution

A new graphic user interface has been developed to gather all the controls and information required by the WARM system. The participation of the mechanics in the design of the interface showed to be fundamental for the success of the subsequent tests. The whole flowchart associated with the Tablet application is shown in Figure 2.

The navigation through WARM has been done in a simple and intuitive way so that the operators do not spend too much time to fill in the maintenance lists. Once the user has logged into the system (Figure 3a), the next screen (Figure 3b) offers the operator the following options are shown:

• Capture multimedia, to take photographs and record videos to describe new incidences found during reparations or prior to begin with them.
• Select a bus to begin/continue the work orders (WO) associated with the user logged in, which opens the corresponding checklist.

The “Maintenance sheet” screen (Figure 4) will load the list of the checks required for the type of maintenance that will be performed on the vehicle (ITV, M1, M2 …). The user can see the fields that have not yet been evaluated with a red background and clicking on them they will be verified and change their background color to white with a green verification symbol. The user must mark the field to record that the revised component works correctly or leave it unchecked to warn otherwise.

In the event that an item is not functional, and additional information needs to be attached, the user has to press and hold the item in the list and the “Notify Incident” screen (Figure 5) will show up. This last screen allows you to upload an image or video previously taken, to capture a new photo, video or audio, or to annotate a comment to enrich the description of the incident. A field that has received an incident notification will appear on the screen with a blue background.

Clicking on the “Send file” button on the “Maintenance sheet” will mean that the maintenance of the vehicle has been completed and it will cause the application to generate a PDF. The complete maintenance information (Figure 6b and Figure 7b) that will be attached to an email, also generated by the application, that will be sent to the server and the person in charge of the vehicle maintenance.

Figure 6 and Figure 7 compare the current reports produced by DBUS mechanics (handwritten document) with the digital reports produced by WARM (shown in pdf format). Notice the picture complementing the annotations of the mechanic.

3.2. AR-Based Solution

To develop the AR solution, an application based on AR with smart headset (such as Microsoft HoloLens, Figure 8) is proposed. These types of AR headset are a new type of hardware that integrates a video camera, a screen and an audio system. The appearance on the market of this type of headset provides a better way of integrating these technologies for non-experts.

AR specific algorithms have been implemented whose objective is to solve the problem of detection and tracking of objects through artificial vision. Due to the fact that objects in industrial environments are characterized by the absence of visual characteristics that can be used in the detection (the usual surfaces are metallic, without patterns or other visual characteristics), two different tracking methods were used to detect the bus and to locate the user respected to the work environment. The geometry and related coordinates that describe our problem are illustrated in Figure 9.

A brief description of the coordinate systems will be given below:

-

W corresponds to the world coordinates system;

-

C corresponds to the camera coordinates system

-

B corresponds to the bus coordinates system with the origin in the license plate;

-

O corresponds to the object coordinates system, i.e., the virtual plane with checklist showed to the user through the headset.

The tracking problem consists in finding a rigid transform function $T_A^B$ which maps points from coordinates system A to coordinates system B. This transform is composed of a rotation and an offset. Rigid transforms are invertible by definition. Therefore, if a transform that goes from system A to B is known, the inverse transform that goes from B to A can be obtained directly by inverting the original matrix. In addition, if three coordinate systems are present (A, B and C) and the transforms from A to B and from B to C are known, it is possible to obtain the transform that goes from A to C by left-multiplying the transform matrices:

$$T_A^C=T_B^C{T}_A^B$$

(1)

The two tracking systems implemented in our approach are the following:

• Detection of the environment and positioning of the operator (spatial mapping): using the 4 infra-red cameras and inertial sensors of HoloLens device, the detection of the most relevant geometric characteristics of the objects of the environment is performed. Through the information obtained with this algorithm, the transformation matrix $T_W^C$ of the device’s camera is estimated and thus, it is able to track the operator in the environment. The MixedRealityToolkit-Unity toolkit (MRTK) has been used as an AR support for this spatial mapping and user interaction. It is a collection of scripts and components designed to accelerate the development of applications aimed at the HoloLens device and Windows Mixed Reality.
• Vehicle detection: the detection of objects must be much more accurate and rigorous than the environment since it is the core of the activity and this depends on the exact positioning of all virtual objects for the operator’s guidance. Due to the large size and poor homogeneity of bus surfaces, it is difficult to position them accurately through a direct detection of the vehicle or its textures. Therefore, it has been decided to develop an algorithm that analyze in real time the images captured by the HoloLens in search of certain fixed elements in all vehicles (license plates). Once the fixed element has been detected, its position can be computed respected to the camera ($T_C^B$). For the image processing, Unity 3D native Vuforia libraries have been used. Vuforia is an AR software development kit (SDK) for mobile devices. It recognizes and tracks flat images (image objectives) and simple 3D objects, in real time.

For the license plate recognition, Vuforia library offers a tool (Image Target Behavior) to recognize and position images in the environment. Through this tool an analysis of the images captured by the HoloLens camera is performed. The algorithm implemented transforms the images of the camera to images with high contrast to apply edge and vertex detection techniques, which it is called “feature points” (Figure 10). Through these feature points, the algorithm makes comparisons to detect known shapes defined by the user like the license plate.

The position of the rest of elements of the vehicle can be estimated knowing its relative position respected to the license plate ($T_B^O$). This can be done in a pre-process calibration stage. Thanks to all of this, it is possible to compute in every single frame the global position of the virtual objects that the system should project to the user.

$$T_W^O=T_B^O{T}_C^B{T}_W^C$$

(2)

Once all the data of the work order are obtained, the algorithms for the spatial mapping and vehicle detection are executed in real time. When all the elements to be verified are located, AR objects are placed in points close to them to guide the operator and assist him in maintenance work. The positioned objects are panels that contain the elements to be verified in maintenance. The panels group elements that have characteristics in common (motor, batteries, control panel, wheels, etc.) and are positioned in an area close to the place where the elements it contains must be verified. Finally, once the entire scene is complete, the operator can interact with AR objects to verify each of the elements of the vehicle that are required. The flowchart associated with the HoloLens application is shown in the following Figure 11:

Each of the panel elements contains a checkbox to inform that this maintenance section has been reviewed and verified (Figure 12). To verify an item, the operator must look at it and by a certain gesture it will be verified. During the element verification process, there are a number of tools in order to attach additional information to the work order. These tools are activated by voice commands for notes dictation and image/video capture.

We would like to emphasize that the contribution of the paper focuses on the implementation of the prototypes of the system in the company’s facilities in an operational environment with real users (TRL7 target) and address the difficulties inherent in the transfer of a technology developed in laboratory (TRL 4) to a real work environment, such as a mechanical workshop. In this scenario, we have tested that workers do not welcome an application when it is too complex. For this reason, we have developed an easier AR application with simple AR objects projections (planes with checklists). As in the tablet-based solution, the designed GUI has been developed to gather all the controls and information required by the WARM system taking into account all suggestions provided by the mechanics.

4. Description of the Experiments

The evaluation of the WARM system has followed the EBSF2 EU project methodology [27], which assesses the performance variation in situations of “do nothing” and “do something” by measuring a sufficient set of Key Performance Indicators (KPIs) [28]. This evaluation framework has been applied to all the case studies of the EBSF2 project and other projects involving multisite demonstrators [29,30]. The methodology defines Validation Objectives (VO) that state general aspects in which the innovation tested is expected to be beneficial. The expected benefits of each VO are quantified through one or more performance targets (PT) that are actually measured through some KPIs. The evaluation method consists of analyzing whether the variation of the KPI values from “do nothing” to “do something” situations meet the expectations for each performance target. To assess whether a technological innovation is worth to retain or not, quantitative target values for each KPI have been set to serve as reference during the evaluation of test results. These improvement thresholds have been decided by the authors to assess the maturity of the innovations, i.e., to measure if the actual performance improvement is sufficient to continue the innovation development from current TRL 6/7 to TRL 9.

For the evaluation of the WARM system, the “do nothing” situation corresponds to the current procedures, which is essentially paper-based handwritten data collection, and it is explained in Section 4.1. For the “do something” situation, two scenarios have been developed: one for the tablet-based solution, explained in Section 4.2, and another for the AR headset-based solution, detailed in Section 4.3.

DBUS operations require that every morning at 6:00 am, a minimum of 103 vehicles of which 2218 m buses and 7312 m buses must be ready to run. The total fleet is composed of 130 buses. Maintenance tasks are organized in three periodical levels (M1 each 10,000 km, M2 each 30,000 km and M3 each 60,000 km), each level contains the previous one. In addition to these three actuations, the PRE_ITV assessment prepares the bus to succeed in the official technical inspection of the vehicle (known as ITV in Spain). In order to reduce the number of days that buses are out of service, the PRE_ITV assessment is made to coincide with one of the three periodical actuations. The garage (Figure 13) is able to service 10 buses simultaneously.

Predictive maintenance work is planned with 7 to 10 days in advance. In addition to maintenance tasks, the daily work at the garage comprises eventually other reparations and actuations due to accidents, breakdowns and incidents occurred during daylight and nightly operations. To cope with this unpredictable workload, maintenance planning is revised every afternoon to guarantee that the needed vehicles to start operations the day after will be available.

These four preventive maintenance tasks (M1, M2, M3 and PRE_ITV) were selected for testing the WARM system. The main purpose of the test was to compare the traditional reporting activity, i.e., handwritten reports, with the WARM-assisted reporting process using the tablet-based solution and the Holoens-based solution. In order to have a homogeneous data volume, the test plan has been oriented to the maintenance of MAN Lion’s City bus models with numbers between 700 and 780. This bus model has the largest number of units in the fleet of DBUS. To compare the WARM system with the system currently used, a series of quantitative and qualitative Key Performance Indicators (KPIs) that represent the most relevant aspects of both maintenance management systems have been defined:

• Dedicated effort in data management: time spent in planning and organizing work orders.
• Dedicated effort in data processing: time spent completing the documentation of work orders.
• Operator perception of workload: subjective perception of different methodologies by maintenance technicians.
• Maintenance costs per vehicle.

A test plan has been developed for each of the defined cases capable of collecting the necessary data for the calculation of the indicators for the evaluation.

4.1. Paper-Based Handwritten Data Collection Test Plan (the Current Procedure)

The collection of quantitative data of the system currently used in the maintenance workshop, has been done manually accompanying the maintenance operator in his working day. For each test, the following data have been taken to calculate the indicators:

• Maintenance date.
• Type of maintenance (M1, M2, M3 or PRE_ITV).
• Bus number.
• Operator in charge of the work order.
• Time to complete the work order.
• Verified items of the work order.
• Total elements to verify.
• Times the checklist has been used.
• Maintenance start and end maintenance time.
• Number of comments added to the work order.
• If maintenance starts on the day or is the continuation of a previous one.

For qualitative data, after performing a maintenance the operators complete an assessment sheet of aspects of their work. Through a series of questions, we can assess the following aspects:

• Workload acceptance.
• Valuation of work orders.
• Valuation notes and attachments.
• General opinion.

4.2. Tablet-Based Solution Test Plan

The data to be collected will be similar to the previous case with the difference that in this case, all will be collected automatically by the device and stored in an internal file. In order to perform the tests with this new device, it is necessary to instruct the operator before starting the tests, for this it is necessary to train worker in the following points:

• Navigation through the application interface.
• Options to activate the different editing functions of the work order (verify item, add multimedia, change report types, quick verification by sections, etc.).
• Attach multimedia files to a work order (audio, video and photos).
• Add comments to document errors or repairs.
• Sending of the finished work order.

Like the previous case, at the end of a work order the operator will be required to complete a similar survey about the application and the device.

4.3. AR Headset-Based Solution Test Plan

Like the previous case, the data to be collected will be similar and will be collected by the HoloLens device and stored in an internal file. Similarly, when introducing a new device in maintenance management, it will be necessary to instruct operators in advance. Being a completely unknown device by the workshop technicians, the instruction should be more accurate. Therefore, it is necessary to explain the following points:

• General description of the test to be performed.
• Comment on what the device and AR is.
• How it fits on the head.
• A reduced simulation of the work to be done is done to explain and practice the basic gestures for handling the device.
• Once the basics have been learned, the maintenance application is executed and all navigation and use through it is explained step by step.
• Finally, the worker is asked if he needs additional information or any extra explanation.

During the whole process, the maintenance technician is accompanied, trying to interfere as little as possible, in order to be able to resolve any questions or problems that arise during the completion of the work order with the new device.

The way to qualitatively evaluate the AR headset system will be similar to the previous cases but adapting some of the questions to the new device.

4.4. Tests Execution and Data Collection

WARM have been demonstrated in DBUS garage, under real operational conditions. Four preventive maintenance tasks (M1, M2, M3 and PRE_ITV) have been the test cases. All systems have been tested in the DBUS maintenance garage. As defined above, the data has been collected manually for the case of paperwork and automatically for the cases of applications for Android tablet and HoloLens AR headset solutions. Informed consents were performed before conducting the study and all data was treated anonymously. Data collection has involved:

• 25 buses of the fleet;
• 23 maintenance operators;
• 3 maintenance managers.

where the evaluation was within-subject and with randomized order.

The data corresponding to the subjective part have been made manually through surveys for each of the cases raised. The surveys are similar for all cases (Figure 14). Two experimental work scenarios are proposed in which tests will be carried out:

• WITHOUT_WARM: in this scenario, the maintenance operator will perform the work as before, this is without the assistance of the WARM system. This will be the control group.
• WITH_WARM: in this scenario, the operator will use the WARM assistance system with Tablet and HoloLens. This will be the test group.

In both scenarios, the same KPIs will be collected during the execution of maintenance work of the same type. The following table shows the structure of the experimental work to be carried out, specifying the validation objectives and the KPIs finally selected for evaluation. Besides, once the bulk of the test data are obtained, it is necessary to debug them in order to obtain the defined indicators and to be able to make a reliable comparison between the proposed steps.

Table 2. Selected KPI in the test and control scenarios to evaluate and validate the system.

# KPI	Validation Objective VO	Performance Target PT	Improvement Target (Threshold)	KPI Description	Units	Formula	Inputs
1.a	Speed up data management	Time improving of data management	10%	Effort dedicated to data management	% workers full time	$\frac{1}{N_d}{\displaystyle {\displaystyle \sum }_{i=1}^{N_d}}\left(W_i\frac{T_i}{t}\right)$	Wi: Workers dedicated to data management on the day i Ti: Total time dedicated to data management on the day i Nd: Number of days t: Duration of the workday
2.a	Reduction of workload	Time reduction of data processing	25%	Effort dedicated to data processing	% workers full time	$\frac{1}{N_d}{\displaystyle {\displaystyle \sum }_{i=1}^{N_d}}\left(W_i\frac{T_i}{t}\right)$	Wi: Workers dedicated to data processing on the day i Ti: Total time dedicated to data processing on the day i Nd: Number of days t: Duration of the workday
2.b			10%	Workload perception from questionnaires	[1,2,3,4,5,6,7,8,9,10] being 1 high workload perception and 10 low workload perception	$\frac{1}{N_W}{\displaystyle {\displaystyle \sum }_{i=1}^{N_Q}}\frac{1}{N_A}{\displaystyle {\displaystyle \sum }_{j=1}^{N_A}}{R}_{ij}$	Rij: Answer to question j of worker i NW: Number of workers NA: Number of questions
3.a	Minimizing operating and maintenance cost	Reduce maintenance costs per vehicle	10%	Cost of maintenance employees per vehicle every 10,000 km	€/(vehicle × 10,000 km)	$\frac{1}{N_b}{\displaystyle {\displaystyle \sum }_{i=1}^{N_b}}{C}_i T_i$	Nb: Number of buses Ci: Cost of maintenance employees for bus i Ti: Maintenance time spent for bus i

summarizes the KPIs to be measured, including the formulae used to calculate them through the data collected and their corresponding improvement objectives.

Some comments about the selected KPIs:

• Effort for data management: for this KPI, the data management is the assignment of work orders to employees. This task is done by managers.
• Effort for data processing: for this KPI, the data processing is the filling of the work orders. This task is done by technicians.
• Staff’s perception of workload: The questionnaire evaluates the comfort to fill the maintenance sheets and report problems. A high value means that the technician finds it easier to work and therefore less workload. All the questions are similar for all scenarios, for this reason we can compare it directly. This task is done by technicians.
• Cost of maintenance staff per vehicle: The data for the cost staff has been provided by DBUS.
• Improvement targets are the threshold values that test results must pass to assess whether a technological innovation is worth retaining.

To evaluate the traditional handwritten reporting method (i.e., without using the WARM system) maintenance mechanics have been accompanied by personnel of CEIT during their activity, who have collected all the necessary data (e.g., times) to produce the requested KPI. Conversely, to evaluate the WARM-assisted reporting process, the WARM device was in charge of collecting data concerning times and other aspects. In this scenario, maintenance workers were also accompanied by personnel of CEIT during their activity, to support them in the use of the tool. After every maintenance task (traditional and WARM-assisted) the survey was conducted to measure subjective perceptions of the workers. At the end of each maintenance work, the maintenance mechanic completed a questionnaire.

5. Results of Experimental Case-Studies

The following

Table 3. Summary of results.

KPI	Units	Test Cases
		Paper	Tablet	AR Headset
1.a—Effort for data management	% workers full time	0.77	0.35	1.67	Less is better
2.a—Efforts for data processing	% workers full time	1.03	0.73	0.73	Less is better
2.b—Staff’s perception of workload	Scale 1–10	7.76	8.44	7.22	More is better
3.a—Maintenance staff costs per vehicle	EUR/(vehicle × 10,000 km)	24.78	24.7	25.09	Less is better

shows the summary of the experiments.

As it can be deduced from the KPIs, clearly the case of a mobile device with Android (Tablet) system is superior to the rest in all aspects. The tablet solution scores the best result in KPI 1.a (effort for data management). This is a consequence of the wide introduction of the tablet technology nowadays. The assignment of a work order to the concerned mechanic is done directly thorough the GIM system, which transmits it to the tablet app seamless. This is much more efficient than the current paper-based procedure that requires to print out a paper copy of every work order and to hand them to the corresponding mechanics. With the tablet solution, all the mechanics get the work orders assigned to them immediately. The mechanics only have to pick-up the tablet from the maintenance office. Once the work is done, the tablet transmits the information to the back office without the need of further participation of maintenance managers. Although similar arguments would also play in favor of the AR headset solution, its result shows poorer performance even compared to the current paper-based procedure. This discrepancy is due to the fact that the AR headset take longer to turn on and the maintenance officer needs more time to verify that the AR headset are ready for the mechanic to use. This fact also penalizes the KPI 3.a, since maintenance workers finally need to spend more time with the AR Headset solution than with the other two.

Both AR headset and tablet solutions perform better on KPI 2.a (data processing efforts), demonstrating the effectiveness and speed of the checklist completion methods implemented by the two technologies.

The tablet solution also scores the best result in KP 2.b, which measures the subjective perception of the workload of the mechanics. AR headset solution scores the lower value, slightly smaller than the current paper-based procedure. This is due to the fact that, today, all the workers in the maintenance shop are familiar with the use of this type of device and do not require any training to use the application. An important issue, apart from the improvement of time and costs, is that the operator perceives a lower workload because the process of documenting work orders is automated and can be detailed with multimedia files unlike the current paper-based system.

On the contrary, in the case of the AR headset solution with AR, a good acceptance by the operators has not been achieved. The device is quite heavy and not very ergonomic for the work environment in which they work. In addition, the process of documenting work orders has been slowed down because this device does not allow to complete them as quickly as the others.

The use of an Android tablet has been beneficial for the good acceptance of this solution, since many mechanics are familiar with Android devices, since they own Android smartphones. Furthermore, mechanics already use their smartphones in their workplace, so they are used to using them safely and cleanly in a very dirty environment. The tablet’s large screen has contributed to its good readability and interaction performance.

Table 4. Final assessment: relative improvements (in %). Negative values correspond to worse values than those obtained with the current procedure (Paper).

KPI	Target	Tablet vs. Paper	AR H. vs. Paper
1.a—Effort for data management	10%	54.5%	−116.9%
2.a—Efforts for data processing	25%	29.1%	29.1%
2.b—Staff’s perception of workload	10%	8.8%	−7.0%
3.a—Maintenance staff cost per vehicle	10%	0.3%	−1.3%

presents the relative improvement or worsening in the KPIs of the tablet-based and AR headset solutions with respect to the current paper-based procedure. The table also includes the expected target for each KPI. It shows that neither the tablet solution nor the AR Headset solution exceeds the target thresholds of all KPIs. However, the tablet-based solution produces much higher improvements in the effort for data management and processing and gets close to the threshold in the staff’s perception of workload. The maintenance cost per vehicle (KPI 3.a) seems to be unaffected by the tested innovations, probably because the worktime reduction (KPI 1.a and 2.a) is very small when compared with worktime required by the reparation tasks.

6. Conclusions

The result of this work has been the development of two solutions for improving the maintenance of bus fleets: one based on Android-based tablet and another one based on Microsoft HoloLens AR headset. The developed solutions are designed as a front-end for interaction with the GIM system. This development is a simplified application of Android GIM that helps fill in the maintenance sheets. This application connects directly through Web Services to the database server.

As a summary of the AR task, different tracking methods (spatial mapping and object detection) have been used to analyze the environment and locate the user respected to the environment itself and the bus. Once the bus is detected and located, it has been possible to include virtual objects in it.

The two solutions have been tested in real operation condition against the current paper-based procedure. Tests have taken place in the garage at the premises of DBUS, which operates the public transport buses in the city of San Sebastian (Spain). Experimental results prove the superior performance of the tablet-based solution in terms of reduction of effort for data management and processing and staff’s perception of workload. Other conclusions are that workers are ready to adopt these tools (tablet-based solutions) that make their work easier. Regarding the additional information collected (videos, photos, etc.), it has been well received by the workshop manager verifying that the multimedia data enriches the information system.

Finally, the analysis of the relative improvement of the proven innovations, presented in Table 4, suggests that the tablet-based maintenance assistance system is worth retaining and therefore seems suitable for further developments to achieve TRL 9.

Taking into account the results and the company feedback, as for a quick and comfortable implementation for operators, future steps will be made towards the implementation of kiosks, instead of tablets. Workers have shown good acceptance of this type of device because it leaves them hands free and they do not need much more to carry out their daily work.

As for further research in AR, future lines should go towards more comfortable devices (it depends on the market) and GUIs that really give real benefits to the user in the form of more functionalities and/or user experience than tablet-based solutions.