Automatic Cube Counting System for the Box and Blocks Test Using Proximity Sensors: Development and Validation

Abstract

The Box and Blocks Test (BBT) is a widely used outcome measure for manual dexterity assessments in neurological rehabilitation. The BBT score is based on the maximum number of cubes that a person is able to displace during a 60s time window. In this paper, a low-cost instrumented system to automatically obtain the number of cubes using proximity sensors is presented. For that purpose, the central partition of the BBT was sensorized, aiming to minimise the employed sensors and minimally alter the physical BBT box. The counting system, connected to the mobile app, allows for the self-administration of the test as users only need to follow the presented instructions. Firstly, the methodology used to automate the test scoring is presented, including the sensors’ description and the prototype design. Then, the obtained success rate in cube counting is shown, with an average of 98% in trials with five healthy users. Finally, the conclusions and future work are shown. The results support the use of automated methods for upper limb assessment, providing more objective results and additional information about user performance.

1. Introduction

Neurological disorders are heterogeneous diseases that affect the central and peripheral nervous system [1]. They involve damage to the brain, spinal cord, cranial nerves, peripheral nerves, nerve roots, autonomic nervous system, neuromuscular junction, and muscles. The non-communicable neurological disorders include epilepsy, Alzheimer’s disease, cerebrovascular diseases including stroke, multiple sclerosis, and Parkinson’s disease, among others.

Neurorehabilitation aims to treat the impairments and problems caused by neurological diseases [2]. Arm and hand function are often impaired in patients with a neurological disease, strongly reducing their ability to perform activities of daily living (ADL). Classical upper extremity (UE) impairments may include deteriorations in gripping force, muscle weakness, or abnormal movement synergies (lack of coordination), among others. Thus, it is of particular concern to assess the extent of UE impairments to generate patient-tailored therapy protocols.

The assessment process aims to understand and quantify the functional impairment level [1,2]. For that purpose, clinicians use standard clinical scales that are specially designed to measure dysfunctionality at different levels. The Fugl–Meyer Assessment (FMA) test is one of the most-used scales in clinical trials [3], and it allows for the assessment of motor functionality, balance, sensations, and ranges of motion for the upper and lower limbs. Focusing on the assessment of UE motor function, the Wolf Motor Function Test (WMFT), the Action Research Arm Test (ARAT), the Motor Evaluation Scale for Upper Extremity in Stroke Patients (MESUPES), or the Unified Parkinson’s Disease Rating Scale (UPDRS) are widely used clinical scales [4]. These types of scales allow for an evaluation of the level of autonomy when performing specific tasks. Other scales that are in widespread use to measure gross manual dexterity and coordination level are the Box and Block Test (BBT), the Nine Hole Peg Test (9-HPT), or the Purdue Pegboard Test (PPT).

Thus, it seems clear that numerous assessment tools are available to clinicians to measure functional limitations in patients with neurological affectations. However, traditional clinical scales usually do not capture the complete spectrum of UE impairments, include inter-operator variability, lack quantitative and sensitive readings, and are often time-consuming to administer. For that reason, there is an active research line regarding the automation of traditional clinical tests to reduce such drawbacks of classical scales [2].

For example, several studies aim to provide the FMA score automatically. Since the FMA is a performance-based test, the FMA score is given according to the manner in which the movements are performed (quality and completion level). Thus, one automation attempt uses inertial measurement units (IMU) to automatically monitor both shoulder and elbow movements and rate the UE-section [5]. A more complete study proposes a framework for automating UE motor assessments that uses low-cost sensors (kinect, IMU, glove) to collect movement data [6]. The sensor data are then processed through a machine learning algorithm to determine the score. Scores obtained by the automatic system are similar to those provided by the traditional FMA without human supervision. A combination of sensors (kinect V2, force-sensing resistor sensors, body-worn sensors) was employed in the study conducted by Lee et al. [7] to assign FMA scores according to a rule-based binary logic classification algorithm. This system exhibited a high scoring accuracy and an 85% reduction in clinician’s required time. Overall, it can be noted that better results are obtained when multiple sensors or technologies are used.

For stroke rehabilitation, an automatic assessment method based on the Wolf Motor Function Test (WMFT) is presented in [8]. Using wearable sensors, the time taken to complete each item in the test is measured and stored automatically. This system covers 7 of the 17 items in the WMFT. Another example is the automation of the ARAT presented in [9]. In this case, the automation approach uses one of the physical elements used in the evaluation procedure (a 7.5 cm wooden cube). Focusing on an evaluation of manual dexterity, a system based on virtual reality and haptic feedback is presented in [10] to automate the Nine Hole Peg Test (9-HPT). The hand coordination level was measured using the grasping force profiles during peg insertion tasks, allowing for healthy and impaired motor skills to be distinguished. However, the self-administration and system usability must be improved, as assistance was required to complete the tasks.

Another relevant disorder for which various automatic systems have been developed to objectively measue functional problems is Parkinson’s disease. For example, a method based on video processing and deep learning techniques is presented in [11] to compute an objective bradykinesia score based on the guidelines of the gold-standard MDS-UPDRS III. This study highlights the potential of deep learning techniques for remote assessment, for instance, via video conference. Similarly, an approach to symptom quantification based on motion data captured by Magnetic, Angular Rate, Gravity (MARG) sensors is presented in [12]. This method shows good results for prono-supination movements, but covers a poor set of movements. Moreover, a vision-based method is presented in [13], where an RGB-Depth camera and a pair of black silk gloves are employed to track the hand gestures. This method presents good results regarding the objective quantification of UPDRS scores; however, the use of hand gloves would be inadvisable for patients with severe hand problems.

It should be noted that the automation of clinical procedures for functional assessment is a relevant topic in neurorehabilitation. Although the test’s administration is an essential aspect of functional assessment, the automation approaches usually focus on score generation, aiming to obtain a more objective metric, a novel score, or extended versions of traditional scores. In this paper, the automation of BBT cube-counting is addressed based on the direct scoring (DS) method [2]; namely, the outcome is obtained by sensing and analysing interactions between the user and the environment. For that purpose, a method based on proximity sensing was used to automatically count the transferred cubes, aiming to reduce clinicians’ manual labour, facilitate the digitalisation of results, and explore the test’s self-administration. The remainder of this paper is organised as follows: Section 2 includes related work regarding the BBT automation. Section 3 presents the methodology and design principles for the proposed system. Section 4 describes the development of the automatic cube counting system, including the mechanical solution and cube counting algorithm. Section 5 summarises the results and trials used to measure the effectiveness in cube counting. Finally, concluding remarks are presented in Section 7.

2. Background

In this paper, the Box and Blocks Test (BBT) was chosen as the study case. Figure 1 presents the BBT’s components and illustrates the use mode for evaluating the gross manual dexterity and coordination. The physical components of the system were a wooden box with two 290 mm wide square compartments and 150 wooden 1 in cubes. A 100 mm high partition was located between the two compartments to separate them. The BBT was administered by a physician by placing the patient in front of the box. The two wooden compartments remained in the mid-line of the patient, and the patient moved the cubes from one side to the other. The purpose of the test was to move as many cubes as possible, one at a time, from one compartment to the other, within one minute. Once the minute is completed, the therapist manually must count the number of transported cubes to calculate the score. The higher the number of blocks, the higher the level of manual dexterity. A complete description of the BBT is presented in [14].

The current literature shows that the automation of cube counting for the Box and Block Test (BBT) has been addressed through various strategies: computer vision, instrumenting objects, or even virtual reality. On one hand, vision-based approaches aim to count the number of blocks without altering the physical setup of the BBT [15,16]. These kind of systems utilise a Kinect sensor to detect the displaced cubes frame by frame. The effectiveness of this method is 100% for up to 30 cubes [17], and additional metrics are provided, such as the partial times of cube motion and hand motion tracking.

On the other hand, other systems propose minimally altering the BBT setup by using wearable sensors on the user’s forearm (wBBT) [18] or embedding sensors into objects used in this assessment (eBBT) [19]. The wBBT system uses five kinds of time series signals, including electromyographic, accelerometer, gyroscope, orientation and orientation Euler [18]. Data from sensors are classified to predict the BBT score, presenting an accuracy in block counting of 99.31% on average. In the case of the eBBT, the BBT box is sensorized by embedding a triaxial accelerometer in the blocks, and the central partition integrates infrared sensors to detect the hand motion [19].

Finally, another strategy is the use of gaming technology to build a virtual environment, which can be used to measure the user’s interaction. Various virtual versions of the BBT have been piloted with healthy subjects [20,21], stroke survivors [22,23], and patients with Parkinson’s Disease [24]. These studies demonstrated that virtual BBT is a valid tool to assess manual dexterity, and the main advantage of virtual environment is its ability to promote a controlled interaction regarding the self-administration of the test [25].

There have been several attempts to automate both the scoring and the administration of the BBT. However, although automated systems have shown promising results, there is still room for improvement, since a single technology can not cover all facets (proper data capturing, automatic scoring, and administration) of the assessment [2]. For example, the strength of automated systems based on a VR environment is the self-administration of the test, but the lack of touch-based feedback is a limitation. Instrumented systems have the advantage of reliable data capturing, with the handicap of increased costs and alterations in the layout of the physical test. Vision-based approaches do not alter the physical setup of the test, but the automatic scoring is influenced by external and difficult-to-control perturbations (occlusions, light conditions, etc.). This paper focuses on developing an automatic method for electronically registering the BBT scoring with a sensorization that minimally alters the BBT box, retains the touching feedback, and promotes self-administration.

3. Methodology

Since the principal outcome of the BBT is the total number of transferred blocks, a frequent research goal was to obtain the total block count automatically. However, the majority of systems do not address automatic administration. As was presented previously, different techniques have been used to automate the cube counting, such as computer vision, virtual reality, or embedding sensors into the objects used in the assessment. In this paper, we propose a method based on this latter idea (instrumenting the objects), but aiming to (1) minimise the sensors used; (2) minimally alter the physical BBT box; (3) provide simple instructions for the self-administration of the test through a specially designed mobile app.

For that purpose, the movement dynamic during the transference of cubes by a healthy user was analysed to identify the best automatic method to detect and count the cubes. As shown in Figure 1, the cubes are released from the top of the central barrier, usually falling into the box compartment by the middle or near the barrier [26]. Note that, depending on the mobility restriction level, cubes could be moved in different ways, and its influence on the proposed method is beyond the scope of this study.

Hence, this paper proposes a method using proximity sensors in the central barrier to detect the cubes while falling. Note that the sensorization of the central partition does not alter the traditional BBT performance. Figure 2 illustrates the proposed system for automatic cube counting and the digitalization of results. Since providing digital results is a relevant feature to distinguish the novel system from the classical one, the proposed system automatically stores the collected data on the cloud via an app running on a smartphone, storing the user records and associated progress during the weeks of treatment.

The proximity sensors suitable for this application must comply with: (1) a compact size to be embedded in the BBT central partition and (2) an adequate detecting range to cover the compartment area.

Proximity Sensors

The proximity sensor SI1143 (Modern Device, 2022) was chosen for this application [27] because it is suitable for light-sensing applications, such as gesture sensing, ambient light measuring, or proximity motion sensing. This proximity sensor includes photo-diodes and driver circuitry for three LEDs in a compact board (32 mm × 32 mm).

Figure 3a shows the development board, including the three LED infrared emitters and the receptor device. LEDs were arranged in an equilateral triangle-shaped layout, occupying each vertex, and the light receptor was placed at the centre of the triangle. Each LED can be driven independently, generating the approximate detection volume of a cone for each LED (see Figure 3b).

The effective proximity detection ranged up to 20 cm for small moving objects. According to the datasheet, the sensor could detect a static object up to 50 cm away under optimal conditions. However, the detection range was reduced to 30 cm in the case of moving objects. Additionally, as the sensor measures reflected light, for the case of small objects, the amount of reflected light can be reduced by one-fourth, resulting in a considerable object detection difficulty for distances above 20 cm.

Additionally, phase-based sensing can be implemented using the arrangement of the three LED emitters. This method involves looking solely at the raw data from the proximity measurements and the timing of the changes in feedback for each LED [28]. When an object is swiped across two LEDs, the direction of the swipe can be determined by looking at which LED’s feedback is raised first. For the case of Figure 3a, if an object falls vertically in front of the sensor, it is first detected by LED 3, and subsequently by LEDs 1 and 2. That is, the LED activation order depends on the direction from which an object approaches the sensor, allowing for direction from which the object has come to be identified. Thus, a phase-based sensing method is suitable for detecting the cube falling in the BBT.

4. System Development

The physical structure of the BBT box allows for the proximity sensors to be arranged in different zones to detect the falling cubes. One option is to place the sensors in the lateral walls of the BBT compartment. The second option is to include sensors in the central partition. After careful consideration of sensor functionality and experimental trials in the laboratory, testing several layout configurations, the following guidelines were obtained for the final design:

• The effective sensing distance is 20 cm; therefore, a single sensor does not cover the full box width (26 cm) or the diagonal of the compartment.
• Sensors positioned on a different plane (ahead of each other or laterally) produce interference (cross-talking effect).
• There is a dead angle in the field of view of sensors near to the base of the conical detection volume (see Figure 3b).
• Sensors placed in the internal wall of compartments and closer to the corners detect the adjunct lateral wall, altering the signal.
• Raising the position of the sensors to the outside of the compartment reduces the interference caused by side walls.

On account of the above, a suitable sensing configuration based on the SI1143 sensor must incorporate sensors that are: (1) arranged in the same plane and (2) placed in a raised position concerning the compartment. For that reason, the central barrier was selected to host the proximity sensors. This piece rises 10 cm above the upper edge of the compartments, and it also allows for the placement of the sensors without altering the physical structure of the BBT and avoids hindering the development of the test. The following section describes the components and development details of the sensorized barrier.

4.1. Sensorized Barrier

The architecture of the system proposed in this manuscript is presented in Figure 4. In addition to the SI1143 sensors, an Arduino Mega ADK board was used to receive the raw data from sensors via the I${}^2$C protocol [29]. The Arduino board implements the signal processing and the algorithm used for cube counting. Moreover, the Wi-Fi module ESP-O1 was used to enable wireless communication between the sensorized barrier and a smartphone. In this way, the data registered by the sensorized barrier were sent to an APP to automatically store and visualize the result.

This application used three SI1143 sensors arranged in-line on the central partition. The sensors were arranged in such a way that the sensing field of view covers the compartment area, and the sensor layout optimizes the sensing. Figure 5 presents the layout of the sensors and their orientation in the central partition.

On the one hand, the orientation of sensors implies that the LEDs 1 and 2 were aligned at the top and LEDs 3 at the bottom to optimize the cube sensing (see Figure 5a). This disposition allows for a higher optical barrier, with six LEDs (two per sensor), to be obtained for primary operation, and a lower optical barrier with three LEDs (one per sensor) to support the detecting barrier presented above. During the usual cube trajectory, first, the cube would cross the higher barrier and activate some of the LEDs 1 or 2. Then, the cube would cross the lower barrier and activate some LEDs 3. The signals produced by the LEDs allow for the implementation of a method of phase-based sensing, relating the falling cube with a specific LED activation sequence.

On the other hand, the sensor layout aims to cover the majority of the compartment area and reduce interference. As previously mentioned, the sensors’ field of view has a death angle. Therefore, it is necessary to reduce these blind spots to avoid a cube not being registered if it crosses over this zone. Based on empirical trials, a distance of 7.5 cm between sensors reduces the blind spots. Hence, it is possible to cover the necessary area to detect the cubes used in the BBT with three SI1143 sensors. Moreover, these sensors were placed at a height of 8.5 cm from the partition base. Thus, the total height of the sensors was 16 cm from the base of the compartment (see Figure 5b). This reduces the interference generated by adjacent sensors because the farthest wall and the base are in a low-intensity infrared zone. As shown in Figure 5b, the areas highlighted in a light red colour illustrate the zone with limited sensing, and the area marked in a red colour represents the high-detecting zone. Thus, the areas farthest from the sensor (distances greater than 20 cm), as the external corners of the BBT compartment, have low capacity for cube detection. This error could be tolerable in people with no severe motor restrictions because, during the expected development of the BBT test, the cubes would not fall directly through that area but would first cross an area that is closer to the central partition and, consequently, the sensors. Note that assistance with cube counting would be beneficial for clinicians in the case of patients with higher dexterity because, the higher the dexterity, the more cubes are transferred.

4.2. Cube Counting Algorithm

The automatic cube detection process using the SI1143 sensors has three components: (1) compartment recognition, (2) sensor signal processing, and (3) cube detection algorithm.

4.2.1. Compartment Recognition

As described in Section 2, to administer the BBT, one of the compartments must be empty prior to starting the assessment. The user must transport as many cubes as possible to the empty compartment from the opposite compartment. Therefore, the sensing system focuses on the empty compartment. This means that, at the beginning of the test, the unique detectable objects are the compartment itself, as illustrated in Figure 5b. Since part of the compartment is inside the sensors’ detection range, it is necessary to quantify the sensor’s readings due to the BBT box or other detectable elements.

For that purpose, the sensorized barrier gathers data from the surroundings for 200 ms to scan the static sources of interference, such as the lateral walls, the bottom of the empty compartment, or another possible nearby objects such as the user’s body. Once the readings from the detectable objects are captured, they are discarded using a Foreground Suppression (FGS) method [30]. Thus, the detection range was calibrated according to baseline interference sources to avoid including static errors from the compartment surroundings during cube detection.

The raw data for sensor readings derive from an analog-digital converter (ADC). The averaged and maximum values of ADC readings form the basis for the foreground suppression method. The averaged readings were used set a baseline threshold, which was improved through the error peak and a correction factor (see Equation (1)). Namely, the threshold value was increased somewhat to ensure the baseline perturbations were discarded. The correction factor was empirically obtained for indoor ambient light levels using the light ambient measurement given by the SI1143 sensor, which has photodiodes that can measure both visible and infrared light. The quotient between these latter results provided the correction factor.

$$AD{C}_{FGS}=AD{C}_{avg}(\alpha +e)$$

(1)

In Equation (1), $AD{C}_{FGS}$ is the final threshold value for the foreground suppression method, $AD{C}_{avg}$ is the averaged value from the ADC sensor signals, $\alpha$ is the correction factor that depends on the environmental lightning, and e is the error defined as the relationship between the averaged and maximum values of the ADC.

4.2.2. Signal Processing

Two principal disturbances appear in the signal obtained by the sensors during typical system operation: ambient noise and interference from other sensors.

Sensor readings from were roughly constant in a static environment, with a measurement of around $375\pm 15$ ADC units. However, the infrared light emitted by adjacent sensors may affect the measurements since the sensors are working in a semi-closed area. This interference is similar to the signal produced when a cube is detected in the central area of the BBT box, resulting in false positives.

Although the above disturbances could be reduced by the FGS method, a simple exponential smoothing (SES) [31] was applied to the native output of the ADC to improve its measurements. The SES is a time series forecasting method for univariate data, without a trend or seasonality. The basic idea of this model is to assume that the future will be more or less the same as the (recent) past. This model requires a single parameter, denoted as alpha ($\alpha$) or the smoothing coefficient, which determines how much importance is given to the most recent demand observation. This model is mathematically represented in Equation (2):

$$AD{C}_i=AD{C}_{i-1}+\alpha (AD{C}_{real}-AD{C}_{i-1})$$

(2)

where $\alpha$ is the smoothing factor, $AD{C}_i$ is the value of smoothed ADC measurement, $AD{C}_{i-1}$ is the value of the previous smoothed ADC measurement, and $AD{C}_{real}$ is the current value of the ADC output. For this application, the smoothing factor was empirically established to be 0.8, and the averaged threshold given by the FGS method was used to initialize the SES model.

4.2.3. Cube Detection Algorithm

Figure 6 illustrates the flowchart for the cube counting algorithm and details the process used to detect a cube using phase-based sensing. Note that this process was applied to the ADC readings from each LED of the SI1143 sensor.

When a cube is thrown into the compartment, it follows a straight trajectory from the hand to the base of the compartment, obtaining a signal from the ADCs that is used to draw a bell-shaped curve (see Figure 7). Three clear parts can be distinguished: rising edge, maximum peak, and falling edge. The first part shows a pronounced increase in the signal values, indicating that an object enters the detection area from one of the flanks. This rising edge is detected when the ADC readings are above the $AD{C}_{FGS}$ parameter obtained by the FGS method. Next, a single maximum point of the curve is obtained. The presence of several peaks indicates that the object has changed trajectory within the sensor detection field. Finally, a downward phase is obtained in the curve, which is symmetrical to the upward phase. This falling edge finalises when the ADC readings reach the $AD{C}_{FGS}$ parameter. Considering a single LED, this is the simplified process for cube detection and registering.

However, to complete the cube identification process, the layout of the LEDs presented in Figure 5a should be considered. It can be seen that a cube crosses top-down. Therefore, a cube first concurrently produces rising edges in LEDs 1 and 2, and then produces a rising edge in LED 3. In other words, the bell-shaped curves in LEDs 1 and 2 are roughly aligned each other, but shift regarding the bell-shaped curve in LED 3. When using this approach, a cube is preliminarily given as valid.

Finally, an additional condition that must be fuilfilled before determining a cube as valid is that the cube first crosses one LED in the higher line sensing and then one LED in the lower line sensing. This condition also helps to reduce the effects of cube rebound. At the beginning of the test, the effects of the cube rebounds are null because the compartment is empty. However, as time passes, the cubes accumulate in the compartment and rebound each other. A user with high dexterity can move more than 80 cubes in the allowed BBT time window, so cube stacking could produce some false positives during cube counting.

4.3. App for Smartphone

A mobile application (App) for a smartphone was developed to make the use of the sensorized barrier more intuitive and store the BBT scores automatically (see Figure 8). The application has a client–server architecture, where the client part is the mobile device, and Firebase is the server part, which hosts the database.

The Firebase Realtime Database (FRD) [32] is a cloud-hosted database from Google LLC. A relevant functionality is that data persist locally, and even while offline, real-time events continue to fire, providing the end-user with a responsive experience. When the device regains connection, the FRD synchronizes the local data changes with the remote updates that occurred while the client was offline, automatically merging any conflicts.

The app was developed using the Android Studio IDE and the design aimed to cover functional requirements, such as user management, an easy-to-use interface, an intuitive display of BBT test results, and automatic result storage. The application communicates with the sensorized barrier by employing the ESP-01 Wi-Fi module, which is built on the ESP8266 micro-controller. For proper operation of the ESP-01 module, the ESP8266 microprocessor driver (Generic ESP8266 Module) and library (esp8266wifi.h) must first be imported. It is also necessary to include the library (FirebaseArduino.h) to connect to the server and to define the host address and the token that will allow for authentication and the execution of read and write actions in the FRD database. Finally, the name and password of the Wi-Fi access point to which the module is to be connected are entered.

Regarding app navigation, if the user does not have an active account on the mobile device, the app displays a login screen where the user must be correctly identified. If the user does not have an account, one can be created on the registration screen. Once inside the app, the user goes to the “HomePatient” screen, which has two buttons: the first one allows for the consultation of previous BBT records, and the second button allows for the BBT test to be launched. In the BBT launching screen, a button is used to start the BBT counting system. This button sends a command to the sensorized barrier to start the counting process. This screen also displays the time in a chronometer to reach the normative 60 seconds of the BBT.

5. System Validation

The implemented system is presented in Figure 9. The central partition was replicated using 3D printing to house the three SI1143 sensors according to the designed layout. In this prototype, only one lateral face of the central partition was sensorized because the main goal was to evaluate the cube counting performance of the proximity sensing-based method. Thus, Figure 9 shows that the Arduino board is not yet embedded in the central partition.

5.1. Experimental Protocol

The feasibility of the automatic cube counting system was evaluated through several trials in the laboratory with healthy users. Firstly, the experiments aimed to quantify the absolute success rate for automatic cube detection when transferring the 150 cubes, without the normative time window limit. Secondly, an additional goal was to identify the limitations in cube detection according to the zone in which a cube falls. Finally, an additional goal was to conduct a preliminary evaluation of the user’s experience with the whole system (sensorized barrier and app). The participants’ appreciation would be useful when preparing pilot trials with patients with neurological disorders.

Pilot trials were conducted at the Assistive Robotics Laboratory of the Universidad Carlos III de Madrid (UC3M). Five young users without reported mobility problems in the upper extremity participated in this study. The demographic data of participants are summarised in

Table 1. Demographic data of participants.

Variable	Data
Age †	22.6 ($\pm 1.52$)
Sex (male/female)	3/2
Dominant side (right/left)	4/1

^† Mean (±SD).

.

System settings are the same as those one shown in Figure 9. Trials were carried out on different days in the same week. The individuals were encouraged to perform the test with their more dexterous hand (dominant). Note that participants were encouraged to move cubes at different hand speeds, as intuitively defined by the participant. Hand speed was not measured. At the end of each stage, an evaluator proceeded to manually count the total of displaced cubes. The manual counting was compared with the automatic counting to quantify the system’s effectiveness. Note that participants tested the traditional BBT setup (including the no-sensorized barrier) to familiarise themselves with the test’s development. This was useful when identifying the differences between both systems. The following sections present the results of the trials.

5.2. Results

5.2.1. Effectiveness in Automatic Cube Counting

Several trials without time limits were carried out to determine the effectiveness of automatic cube counting in the proposed system. The cubes were individually transferred in batches, and the batch size increased by ten cubes for each repetition until reaching 150 cubes. Therefore, each participant performed 15 batches, beginning with ten cubes, and progressively increasing by ten cubes after each attempt. At the end of each batch, the system was reset (compartment emptied). In consequence, a total of 75 cube-transferring batches were used in this feasibility study for all participants.

Figure 10 shows the number of cubes detected by the proposed automatic system during trials, presenting an excellent and roughly constant success rate. The system showed 100% accuracy when 30 cubes are transferred, while an effective detection rate of 97% was obtained for more than 30 transferred cubes. Overall, the success rate for automatic cube counting was 98.22% on average (percentage of correct recognition).

According to the normative data of the BBT [14], the average number of transferred cubes for females was $78.4\pm 10.4$ and for males, the number was $76.9\pm 11.6$. Hence, it seems that the proposed system could be suitable for the automatic detection of cubes in the usual operating range of the BBT. However, more trials are needed to support this approach and evaluate the performance of the cube counting algorithm in the normative one-minute window.

5.2.2. Detection Performance by Compartment Areas

Since the proposed method is based on proximity sensing, it is relevant to quantify the cube detection performance according to compartment zone. For that purpose, the total compartment area was divided into 13 subareas of interest to identify the limitations in automatic cube detection in such subareas (see Figure 11a). These subareas were defined according to the position of sensors and the expected cube trajectories observed in previous work [16,17]. Ten cubes were dropped from a height of 20 cm in each area of interest in freefall.

The results of these trials are shown in Figure 11b. As noted, the better performance (marked in green colour) was obtained near the sensorized barrier (areas from 1 to 7) and in the middle of the compartment, while the worst performance (marked in red colour) was identified in the most distant corners. The cube detection success rate was above 98% in the surroundings of the sensorized barrier and above 96% in the vertical central band (subareas 8, 9, and 10) of the compartment. Note that, in front of sensors, the system detected 100% of cubes. Howeve, the furthest vertical band (subareas 11, 12, and 13) presents a detection success of approximately 4%.

5.2.3. Preliminary Usability Testing

A secondary interest of this study was related to the user experience. Participants were invited to fill in a questionnaire to assess the usability of the automated system and the app. Questions were classified into two categories: app utility and use mode of the sensorized barrier. The features were evaluated by each user expressing their opinions through satisfaction scores ranging from −2 (strongly disagree) to +2 (strongly agree). Regarding the number of users required for a proper usability evaluation, five is a proper sample size for usability testing [33,34].

Overall, the user experience with the sensorized barrier and app was satisfactory, and the results are summarised in

Table 2. Results of the usability questionnaires.

N°	Question	Mean	Mode	SD
App utility
Q1	Could the app help to administer the test?	1.4	1	0.55
Q2	Is the automatic data storage useful?	1.8	2	0.45
Q3	Is the app design proper to this application?	−1	−1	0.71
Q4	There was no troubles when connectivity	0.2	0	0.84
Q5	Can you navigate intuitively through the screen?	1.2	1	0.45
Use mode of sensorized barrier
Q6	I do not require assistance to use the system	1.6	2	0.55
Q7	Automatic system was confusing to use	−1.4	−1	0.55
Q8	I consider the system useful	1.6	2	0.55
Q9	I prefer not to use the sensorized method	−1.2	−2	0.45
Q10	Have the system taken a lot of effort from you	−1	−1	0.71

. The best results were obtained regarding the use mode of the automatic system. Participants highlighted its simplicity to use (Q7), and most of them found it handy (Q8). The majority preferred the automatic system to manual counting (Q9). The app’s utility was considered positive, but it was shown to require some improvements. The intuitive navigation (Q5) and simple layout (Q3) were moderately accepted. Participants pointed out that the graphic design requires improvements and the policies of universal design need to be included. However, the function of automatic data storage (Q2) in the cloud was considered highly valuable. There were no connectivity issues (Q4) because the system was prepared to auto-link.

6. Discussion

At present, the automation of traditional clinical tests for functional assessment is a recurrent topic in neurorehabilitation [35]. This is understandable, as the traditional assessment procedures have some drawbacks, such as inter-operator variability (subjectivity), their being labour-intensive and requiring manual administration, and the lack of digital results. The development of automated assessment systems (AAS) may help to reduce the above limitations of classical functional evaluation.

Hence, this study focused on validating an instrumented system based on proximity sensors to automatically score the BBT test, which is widely used in manual dexterity assessments. Since the main goal is to evaluate the feasibility of the proposed cube counting system, the instrumented system was piloted with five healthy users. A total of 75 cube counting trials were performed. On this basis, the present study presented three relevant findings: (1) the success rate in automatic cube detection using proximity sensing with the proposed layout is elevated, (2) the sensorization of the central partition seems adequate, not altering the usual test’s administration, and (3) the use of a mobile app facilitates data management.

The success rate for automatic cube counting up of to 150 cubes is elevated, with an average correct recognition of 98%, better than vision-based methods [15,16,17] and similar to other instrumented methods [19]. Trials were performed on different days of the same week but at different hourly segments. In the morning, the room was illuminated by natural ambient light and, in the afternoon, by conventional artificial indoor light. Consequently, the environmental light conditions were different during the performance of the trial. The cube counting esults suggest that ambient or artificial light does not significantly influence automatic cube detection. This is a relevant feature, as other systems aiming to automate BBT scoring reported the high influence of environmental light conditions because they used computer vision methods [15,16,17]. However, the detection rate slightly varies in the proposed system according to dynamical factors such as the speed of cube displacement or cube stacking. However, the variation seems to be systematic for 150 cubes and could be discarded.

The results demonstrate that sensorization of the central partition of the BBT does not influence the typical test administration process. During trials, participants had no reported difficulties displacing the cubes using either classical or sensor-based central partition. Note that participants used the classical BBT setup to familiarise themselves with the method. However, one of the identified limitations of this proximity sensing method is that the rules of the BBT, such as not displacing more than one cube at a time or [14], cannot be checked. Another concern is the inability to distinguish between cube and hand, resulting in possible false positives. To deal with these aspects, the Wizard app was designed to include a brief video tutorial to instruct the user throughout the BBT stages, promotin automatic test administration. No execution time score is provided on the app screen to avoid cheating or user frustration while performing the test.

Finally, it seems that using a mobile app is a viable approach to help with data management without increasing the labour of patients or clinicians. It was suggested that storing the collected data in an online database such as Firebase is suitable for digitalising the BBT scores. Previously, the smartphone was configured to obtain access to this BBDD using the user profile in anonymous way, storing and retrieving the scores of all the previous sessions. Regarding usability, participants have not reported discomfort with the app. All of them were able to easily navigate the screens and visualise the results after the trials. Although the user experience was gathered from healthy users, the preliminary results would be convenient when enhancing the app design and preparing for future clinical trials featuring patients with neurological disorders.

Despite the good results obtained for cube counting, the system can be improved in different ways; for example, improvements with the detection issues and the self-administration of the test. Firstly, the SI1143 sensor was used in this paper due to its small size and easy-to-use features. However, the reduced sensing range was a drawback when attempting to cover the furthest areas of compartments. This issue could be solved by using another sensor with a more extensive detection range, while maintaining the reduced board size. Secondly, as the system’s focus is automatic cube counting, assistance with the test’s administration was limited to brief user instructions regarding the test’s rules and stages via the app. Therefore, a remaining issue is how to include more strategies to address the automatic administration of the test, which would be a relevant aspect in tele-rehabilitation. The major concern may be the assistance level that a user can demand. Namely, the preliminary stages of the BBT evaluation consist of obtaining one empty compartment, which involves grasping capabilities. However, depending on the affectation level, a user cannot prepare the test for administration. Thus, self-administration is a relevant unsolved problem when attempting to obtain fully automated assessment systems.

In summary, including sensors in the central partition seems to be the best strategy to automate cube counting without disturbing the normal assessment procedure. This automatic counting system, complemented by the app, is a step towards obtaining a feasible tool for the assessment of UE functional impairments in tele-rehabilitation, whose relevance was highlighted after the COVID-19 pandemic [36,37].

Limitations

This study has some limitations. Firstly, the results presented in this paper were obtained with a few, healthy, young participants. Therefore, the conclusions cannot be directly applied to patients with neurological impairments. The level of mobility restrictions must be considered in future trials to identify the most suitable target population for this application. Secondly, the usability tests performed in this article offer preliminary insights. The goal was to gather the user experience of participants to set a baseline to prepare for future clinical trials. Aspects such as participants’ familiarity with technological devices and smartphones must be considered. Finally, the current layout entails that it is not possible to track hand motion nor distinguish between hand and cube. Therefore, false positives could be introduced in the final counting. Including additional sensors at the partition’s top or combining the proposed system with vision-based systems could help to reduce this limitation.

7. Conclusions

This paper presents a method based on low-cost proximity sensors, which can automate cube counting during manual dexterity assessments using the BBT. The design principle is to minimise the number of sensors without obstructing the typical BBT administration. The sensorization of the central partition is a feasible solution that fulfils these requirements. The effectiveness of automatic cube counting using the sensorized barrier was excellent, obtaining an average success rate of 98% for up to 150 cubes with healthy users. This supports the use of proximity sensing to automate BBT scoring. Additionally, the use of a mobile application was proposed to digitalise the results and promote the self-administration of the test, showing positive acceptance.