Gesture Vocabularies for Hand Gestures for Controlling Air Conditioners in Home and Vehicle Environments

Abstract

With the growing prevalence of modern technologies as part of everyday life, mid-air gestures have become a promising input method in the field of human–computer interaction. This paper analyses the gestures of actual users to define a preliminary gesture vocabulary for home air conditioning (AC) systems and suggests a gesture vocabulary for controlling the AC that applies to both home and vehicle environments. In this study, a user elicitation experiment was conducted. A total of 36 participants were filmed while employing their preferred hand gestures to manipulate a home air conditioning system. Comparisons were drawn between our proposed gesture vocabulary (HomeG) and a previously proposed gesture vocabulary which was designed to identify the preferred hand gestures for in-vehicle air conditioners. The findings indicate that HomeG successfully identifies and describes the employed gestures in detail. To gain a gesture taxonomy that is suitable for manipulating the AC at home and in a vehicle, some modifications were applied to HomeG based on suggestions from other studies. The modified gesture vocabulary (CrossG) can identify the gestures of our study, although CrossG has a less detailed gesture pattern. Our results will help designers to understand user preferences and behaviour prior to designing and implementing a gesture-based user interface.

1. Introduction

As modern technologies become increasingly incorporated into everyday life, mid-air gestures have become a promising input method in the field of human–computer interaction (HCI) as mid-air gestures represent a natural human–computer interaction with no need for physical interfaces [1]. For instance, gesture acquisition technologies (e.g., Ultraleap Leap Motion Controller, Ultraleap Stereo IR 170 and Microsoft Kinect) and gesture-supporting software facilitate gesture acquisition [2,3], analysis [4] and recognition [5] and thus comprise a new generation of touchless user interfaces that promote gesture interaction for various environments (e.g., home [6,7,8] and driving [9,10,11,12]).

The air conditioner (AC) is a device that can be found in offices, shops, homes and vehicles. The common functions of AC in cross environments are turning on/off and changing the temperature (increase/decrease), the air direction (horizontally/vertically), the fan speed (speed up/slow down) and the mode. Many studies have examined the gestural user interfaces used to manipulate ACs either at home [8] or while driving a vehicle [9].

Mid-air gestures can be defined by software researchers, vendors, designers and hardware manufacturers; however, the actual needs of end users are often ignored. For instance, in studies [11,13,14,15], the researchers defined gestures to control some devices, including AC, and asked users to remember and perform those gestures. The findings of those studies showed that the gestures introduced by experts/designers were not suitable for the actual users. In other studies, the gestures were reported to be difficult to learn [16] and remember [17]. Elicit studies provide an effective approach to defining the user-preferred gestures to control a certain device or a group of devices. In this method, actual end users are involved in the gesture development lifecycle, resulting in an improvement of the accuracy of gesture recognition and achievable user satisfaction [18]. For instance, actual end users were prompted to elicit their preferred gestures in order to design a more efficient user interface in [19]. In another study, the end users were individually involved in performing the desired gestures toward a specific task, also called a referent, when interacting with the device’s user interface. The top employed gestures for each task were identified and assigned to control a certain task [9]. Other studies optimised the selected gestures for more suitable results. They applied some usability tests on the obtained gestures [20] or attempted to gain the highest number of agreed gestures among users assigned to corresponding tasks [19,20,21]. Indeed, elicit studies have been a useful method in a wide variety of interaction application domains such as mobile phones [22], manipulating remote 3D objects [23] and gaming [24].

Although gestural user interfaces and technologies have had a remarkable development, possible improvement can be made in terms of formally analysing and evaluating these systems’ user interfaces for AC systems in different environments. In this paper, we propose a user-elicited study to define an efficient gesture vocabulary (HomeG) for an AC system at home. Additionally, we propose another gesture vocabulary (CrossG) for controlling an AC system in the home as well as in a vehicle. The variety and complexity levels of gestures included in a gesture vocabulary can provide the basis for developing a gesture-based user interface. A sufficient understanding of gesture forms would lead to an increase in our ability to provide a benchmark used for gesture-based interfaces for AC systems at home and in vehicles.

We conducted the study with 36 participants. Each participant was required to control a home AC unit using 12 mid-air hand gesture commands. Most of these commands had been previously evaluated for in-vehicle AC by [9,10]. Following the gesture taxonomy introduced by [10] with modifications, we defined a gesture vocabulary for the 12 commands and compared them with those proposed by [10].

This paper contributes the following to the area of gesture-based user interface research: (1) two user-defined gesture sets: one for operating the AC system at home and another for operating the AC system at home and in a vehicle together, covering 12 AC functions, and (2) an understanding of the influence of environment on the preferred gestures and thus on the gesture vocabulary. Our results will help designers implement better gesture-based user interfaces.

2. Related Works

Our work primarily concerns research related to gesture-based interactions for controlling the AC in a home or vehicle. Thus, this section focuses on work in these two environments.

2.1. Gestures to Control Air Conditioners at Home

Homes typically can have several controllable devices (e.g., AC, televisions, lights and multimedia players). These devices often share some features or functions, which make it possible to have one gesture simultaneously control more than one device. As a result, the same gestures might operate different devices even though the user only needs to control one specific device [25]. Thus, some gesture interfaces recognise the desired device first and then operate it. That is, the user specifies the target device by employing a registration gesture or a device-selection gesture, and the user then performs the gesture (i.e., the command gesture) that controls that device [8]. It is also possible to use one hand to select the device and the other hand to control it [20]. Alternatively, the user can select the target device from a list shown on a pair of smart glasses and then control the selected device [14]. One study [25] went further by requiring the user to perform a specific unique gesture for authorisation, called a user security gesture, that authenticated the user to use the system prior to selecting the target device and then controlling it. However, [26] did not consider the registration gesture or security gesture in the study. They only required the user to stand in front of the target device to begin controlling it.

Table 1. A comparison of hand gesture studies based on the environment, whether AC is the only studied device, and the AC commands. M: multi devices [M] partially supporting multi devices; S: specific device; •: supported.

Authors	Environment	Specific/ Multi Devices	Cross Environment	On/Off	Left/Right	Up/Down	Fan Speed +/−	Next/Pred Mode	Temp +/−
[8]	Home	M	No	•					•
[25]	Home	M	No	•			•	•	•
[20]	Home	M	No	•					•
[14]	Home	M	No						•
[26]	Home	[M]	No	•					•
[11]	Vehicles	M	No						•
[27]	Vehicles	M	No	•					•
[9]	Vehicles	S	No	•	•	•			•
[28]	Vehicles	M	No	•
[10]	Vehicles	M	No	•	•	•			•

reviews hand gesture studies based on the environment, the studied device and the AC commands.

Home AC can have many different commands. The most commonly studied commands were turning the AC on/off and adjusting the temperature (see Table 1). Further AC functions were discovered in [25], such as adjusting the fan speed up/down and changing the mode (heat–cool; cool–heat). However, these studies ignored or did not include “activating/deactivating the wind direction” in the commands studied. To perform those commands, the user can use one or both hands when generating the gestures. A study conducted by [14] forced the user to use only their right hand, particularly the thumb of the right hand, to control the AC in a smart home environment. Other studies left the user free to use either hand or even both hands together when producing gestures [8,25,26]. In the case of simultaneously using both hands, the generated gesture can be identical (both hands performing the same gesture) or random (e.g., one hand is pointing at the AC while the other hand moves upward). However, the number of hands used does not matter when classifying the gestures as long as they match some factors, such as shape and direction [20]. Therefore, the gestures produced by either one hand or two hands can be grouped together.

2.2. Gestures to Control Air Conditioners in a Vehicle

In-vehicle AC has similar commands to that of home AC, but there are minor differences. An in-vehicle AC has at least three panels: the driver panel, the passenger panel and the middle panel. The driver usually controls the driver and middle panels. These panels have horizontally and vertically adjustable swings. The driver can adjust the swings, but they will remain inactive until the driver adjusts them again. Adjusting the AC panels using hand gestures was discussed in [9,10]. Other AC commands were discussed as well (see Table 1). A major difference between controlling the AC at home and in a vehicle is that controlling the AC is a secondary task while driving. The hands are typically on the steering wheel, controlling the vehicle, and the free space in which the user can move their hands is not as large as it is at home. Additionally, performing a secondary task might district the driver’s attention, which leads to poor driving safety [27]. Although a gesture user interface in a vehicle reduces the visual demand [21,28] and driving errors [10] compared to other interaction methods, producing gestures that are easy to remember and perform for in-vehicle devices might be limited in terms of nature and form.

Similar to the home, vehicles have several devices and user interfaces. Therefore, Refs [10,11,28] investigated a variety of gestures for controlling in-vehicle devices. To distinguish the target device, the user in [28] firstly activated the target device using a gesture that allowed them to control it. Other studies (i.e., [10,11,27]) did not specify gestures for distinguishing the target device. They only studied the suitable set of control gestures for different in-vehicle devices.

Only two fingers (thumb and index) were used in [11], without requiring the driver to take his/her hand off of the steering wheel. In [28], a head-up display was used to inform the driver of the required number of fingers on which hand should be spread from the steering wheel in order to switch on/off four in-vehicle devices. The fingers of the right hand were used to turn on the device, while the fingers of the left hand were used to turn off the device. For instance, the driver could turn the AC on by turning out any three fingers of their right hand away from the steering wheel and turn it off by turning out any three fingers of the left hand away from the steering wheel. Thus, the driver’s hands did not need to fully leave the steering wheel when switching the desired device on or off. In addition to using fingers to derive gestures for in-vehicle AC, full-hand gestures were also used [9,10,27]. Although there was an option to use any hand to produce the gesture in [9,10,27], only one hand was used to generate the gesture as the other hand was on the steering wheel [27] or because the nature of the target task might be easier to perform with one hand than the other (e.g., “horizontally control the air direction of the AC middle panel”) [9,10]. One of the most-cited works in user-defined gestures [19] states that the number of fingers used when performing a gesture does not matter, and gestures of more than one finger can be grouped with gestures performed by one finger. Nevertheless, the elicited gestures in [9,10] (see

Table 2. GestDrive’s gesture codes, types and description, adapted from [10].

Code	Gesture Type	Description
G1	Full hand static pose	Hand pose is held in one location
G2	Full-hand dynamic pose	Hand pose changes in one location
G3	Full hand static pose and move leftward/rightward	Hand pose is held as hand moves leftward/rightward
G4	Full hand static pose and move up/down	Hand pose is held as hand moves upward/downward
G5	Full hand static pose and path	Hand pose is held as hand turns around
G6	Two-finger dynamic pose	Two fingers’ pose changes as hand turns around
G7	Two-finger static pose and move leftward/rightward	Two fingers’ pose is held as hand moves leftward/rightward
G8	Two-finger static pose and move up/down	Two fingers’ pose is held as hand moves upward/downward
G9	Two-finger hold	Static pose with two fingers
G10	One-finger hold	One finger is held in one location
G11	One finger moves left/right	One finger’s pose is held as hand moves leftward/rightward
G12	One finger moves up/down	One finger’s pose is held as hand moves upward/downward

) were interpreted in accordance with the gesture taxonomy that was produced by [9,10,19], differentiated in their gesture vocabulary (GestDrive) between one-finger gestures and more-than-one-finger gestures, even in the case that both gestures had the same direction and shape. In addition, the movement of the gesture was not specified in [19]. In contrast, [9,10] describe whether the movement of the hands/finger was left to right, up to down or turning, and categorised the gesture accordingly. They found that the majority of the gestures were employed using the full hand, followed by gestures employing two fingers and one finger.

2.3. Evaluation of Existing Studies

Table 1 compares different hand gesture studies according to the environment (home (H) or vehicle (V)), the number of devices covered in the study (multi-device (M) or single device (S)) and the studied AC commands. None of the studies examined cross-environment hand gestures. They focused only on one single environment with a limited number of AC functions. This is probably because the gestures they developed were designed to cover AC functions in addition to the functions of other devices. Even when the hand gestures were derived only for vehicles [10], the study did not include all AC functions. However, [10] covered more AC functions than other studies that examined in-vehicle devices. Therefore, in the current study, we focus on hand gestures that are derived from a home environment and compare them with gestures employed in a vehicle environment to determine the possibility of using the same gesture taxonomy for both environments.

3. Methodology

3.1. Participants

Thirty-six university students and young teachers (29 right-handed participants; 9 left/mixed-handed participants) participated voluntarily in the experiment. All participants were male, aged from 18 to 29 with a mean age of 24.49 years (SD = 2.89). All were recruited from the public.

3.2. Selecting Target Functions

According to the evaluation of the related work, 12 AC functions were selected in this study (see

Table 3. AC Tasks.

Task #	Description
Task 1	Turn on the AC
Task 2	Increase the temperature
Task 3	Decrease the temperature
Task 4	Activate the horizontal wind direction
Task 5	Deactivate the horizontal wind direction
Task 6	Activate the vertical wind direction
Task 7	Deactivate the vertical wind direction
Task 8	Speed up the fan speed
Task 9	Slow down the fan speed
Task 10	Change the AC mode (next)
Task 11	Change the AC mode (previous)
Task 12	Turn off the AC

). All these functions were derived from the related work. They can be used to operate the AC system at home and in a vehicle.

3.3. Procedure

Each participant received information about the study. If the participant agreed to participate, then their demographic information (age, gender and handedness) was collected. In the experimental session, the participant stood on a designated point that was located approximately 2 m in front of the AC (see Figure 1a). Although most studies (e.g., [9,15,20]) did not mention the distance between the device and the user, we decided to fix this distance at 2 m to unify the distance for all participants. A green chroma was used to easily extract the gesture for future analysis. The participants orally received a list of 12 tasks—one task after another. All participants received the same tasks in the same order (see Table 3). Participants were encouraged to think about how to apply a mid-air gesture and were then asked to perform their preferred gesture to complete each task. Participants were asked to produce one gesture for each task, using either one hand or both hands. They were allowed to repeat the same gesture for different commands if they wished. The participants put their hand down to indicate that they had completed the task. The researcher then executed that task using the AC’s remote control to make the user experience more real. For instance, if the participant performed the gestures for turning on the AC, the researcher turned the AC on using the remote control, and the participant then waited for the next task, and so forth.

3.4. Data Acquisition and Analysis

Two cameras were used to film every experimental session, with a camera installed behind each of the participants’ hands (see Figure 1b). The filmed sessions were observed and analysed to extract the user-elicited taxonomy to develop a gesture-based interface for controlling a home AC system which can then be transformed to control an in-vehicle AC system. The observations included reviewing the preferred gesture used to perform each task for each participant (see Figure 1c). As is shown in Figure 2, the elicited gestures were first generally observed in terms of the number of hands used to generate the gestures. An identical, both-hands gesture meant that both hands generated the same form in the same direction. Similar to [20], we dealt with such gestures as one gesture, not two. Random, both-hand gestures, however, meant that one hand was used as an activation/deactivation or on/off gesture while the other hand generated the command gestures. In this case, only the command gestures were included in our taxonomy. The elicited gestures were then interpreted in accordance with the gesture taxonomy produced in [4], which is called GestDrive (see Table 2). GestDrive covered most of AC functions discussed in the related work. However, GestDrive showed poor suitability for a home AC gestural interface. Therefore, we proposed a new gesture vocabulary, HomeG (see

Table 4. Gesture code, function, and description for HomeG for controlling home AC system.

Code	Function	Description
FH1	Full hand static pose	Hand pose is held in one location
FH2	Full-hand dynamic pose	Hand pose changes in one location
FH3	Full hand static pose and move leftward/rightward	Hand pose is held as hand moves leftward/rightward
FH4	Full hand static pose and move upward/downward	Hand pose is held as hand moves upward/downward
FH5	Full hand static pose and path	Hand pose is held as hand turns around
FH6	Full-hand dynamic pose and turn	Hand pose changes as hand turns around
FH7	Full-hand dynamic pose and move leftward/rightward	Hand pose changes as hand moves leftward/rightward
FH8	Full-hand dynamic pose and move upward/downward	Hand pose changes as hand moves upward/downward
FH9	Full-hand dynamic pose and move forward/backward	Hand pose changes as hand moves forward/backward
TF10	Two-finger dynamic pose and turn	Two fingers’ pose changes as hand turns around
TF11	Two-finger static pose and move leftward/rightward	Two fingers’ pose is held as hand moves leftward/rightward
TF12	Two-finger static pose and move upward/downward	Two fingers’ pose is held as hand moves upward/downward
TF13	Two-fingers static pose	Two fingers’ pose is held in one location
TF14	Two-finger dynamic pose	Two fingers’ pose changes in one location
TF15	Two-finger dynamic pose and move upward/downward	Two fingers’ pose changes as hand moves upward/downward
TF16	Two-finger static pose and path	Two fingers’ pose is held as hand turns around
OF17	One-finger static pose	One finger’s pose is held in one location
OF18	One-finger static pose and move leftward/rightward	One finger’s pose is held as hand moves leftward/rightward
OF19	One-finger static pose and move upward/downward	One finger’s pose is held as hand moves upward/downward
OF20	One-finger static pose and path	One finger’s pose is held as hand turns around
OF21	One-finger dynamic pose	One finger’s pose changes in one location
OF22	One-finger dynamic pose and turn	One finger’s pose changes as hand turns around
OF23	One-finger dynamic pose and move leftward/rightward	One finger’s pose changes as hand moves leftward/rightward
MF24	Three-finger dynamic pose	Three fingers’ pose changes in one location
MF25	Three-finger dynamic pose and turn	Three fingers’ pose changes as hand turns around

), which showed superior suitability for elicited gestures used to control home AC functions. Nevertheless, it produced a relatively large number of gesture types and thus the user might find it difficult to remember when to use those gestures in the vehicle. Hence, we combined HomeG and GestDrive with modifications to produce another gesture vocabulary, CrossG, to facilitate employing gestures to control the AC when moving between the home and vehicle (see

Table 5. Gesture code, equivalent codes from HomeG, pattern and description for CrossG for controlling an AC system across environments.

Code	Equivalent Code	Gesture Pattern/Type	Description
FH1	FH1	Full hand static pose	Hand pose is held in one location
FH2	FH2	Full-hand dynamic pose	Hand pose changes in one location
FH3	FH5	Full hand static pose and path	Hand pose is held as hand turns around
FH4	FH6	Full-hand dynamic pose and turn	Full hand pose changes as hand turns around
FH5	FH3 + FH7	Full hand static/dynamic pose and move leftward/rightward	Hand pose is held/changed as hand moves leftward/rightward
FH6	FH4 + FH8	Full hand static/dynamic pose and move upward/downward	Hand pose is held/changed as hand moves upward/downward
FH7	FH9	Full-hand dynamic pose and move forward/backward	Hand pose changes as hand moves forward/backward
AF8	TF13 + OF17	Any-finger static pose	Any fingers’ pose is held in one location
AF9	TF14 + OF21 + MF24	Any-finger dynamic pose	Any fingers’ pose changes in one location
AF10	TF16 + OF20	Any-finger static pose and path	Any fingers’ pose is held as hand turns around
AF11	TF10 + OF22 + MF25	Any-finger dynamic pose and turn	Any fingers’ pose changes as hand turns around
AF12	TF11 + OF18 + OF23	Any finger static/dynamic pose and move leftward/rightward	Any fingers’ pose is held/changed as hand moves leftward/rightward
AF13	TF12 + TF15 + OF19	Any finger static/dynamic pose and move upward/downward	Any fingers’ pose is held/changed as hand moves upward/downward

). The general discussion is subsequently addressed.

4. Results and Discussion

With 36 participants and 12 sub-tasks, a total of 36 × 12 = 432 gestures were produced. Most of the gestures were generated from only the right hand (65%), while left-hand-only gestures ranked second, with 27% (see Figure 3a). Both-hand gestures were either identical or random.

4.1. GestDrive: Taxonomy Breakdown for Gestures

Firstly, we analysed the generated gesture for each AC control task in accordance with GestDrive, the gesture vocabulary proposed in [9]. Most of the gesture poses employed a full hand (284 gestures (66%)), then one finger (83 gestures (19%)), followed by two fingers (60 gestures (14%)) and, lastly, three fingers (5 gestures (1%)). The observation of the obtained gestures demonstrated that all gesture types in the vocabulary were found in our gestures. However, we found that the vocabulary efficiently identified 227 gestures which represented 53% of the total gestures (see Figure 3b).

Figure 4 shows the percentage of each gesture obtained in our study and successfully identified by the vocabulary in [10]. The full-hand dynamic pose (G2) gesture type was the most commonly generated gesture among the identified set of gestures (28.6%). The full hand static pose and move up/down (G4) gesture type ranked next, with 18.5%. The least preferred gesture types were the two-finger static pose, the move left/right (G7) and the two-fingers hold (G9).

4.2. Discussion of GestDrive

A total of 205 gestures, which represented 47% of all gestures, were frequently used by participants but not efficiently identified by the gesture vocabulary in [10]. From this perspective, GestDrive demonstrates poor suitability for a home AC gestural interface, although it showed high suitability for an in-vehicle AC gestural interface. GestDrive was not able to identify further poses completed using the full hand, one finger or two fingers. In addition, GestDrive does not include any gesture that is generated by three fingers. Therefore, there is a need to define additional gesture patterns for the unidentified gestures. The current study has extended GestDrive. The following subsection highlights the total obtained gestures, including detailed information about unidentified gestures. This will help us to gain a better understanding of the differences between the gestures that can be used in both environments: home and vehicle.

4.3. HomeG: Taxonomy Breakdown Based on Proposed Gesture Vocabulary

We used the gesture classifications described in Table 4 to demonstrate the new detailed taxonomy (HomeG), which consists of 25 gesture types. The distribution of new gesture types is shown in Figure 5. Full-hand dynamic pose (FH2) was the most preferred gesture, with 15.0% usage. Full-hand dynamic pose and turn (FH6) came second with an employment of 12.0%. Full hand static pose and move upward/downward (FH4) was the third most preferred gesture with 9.7%, followed by full-hand dynamic pose and move leftward/rightward (FH7) with 8.3% and one-finger dynamic pose and turn (OF22) with 8.1% and full-hand dynamic pose and move up/down (FH8) with 7.9% were the fifth and sixth most preferred, respectively.

A variety of gesture criteria were assigned to each task, ranging from 9 to 14 criteria with an average of 11 criteria (see Figure 6). Changing the AC mode to the next option was associated with the largest number of gesture criteria (14 different gesture types), whereas the fewest gesture types were used to activate the vertical wind direction.

The gesture distribution for each of the 12 AC tasks is shown in Figure 7. The dichotomous tasks are shown next to each other. For turning on the AC, FH2 was the most preferred gesture with 44% usage, followed by TF6 with 17%; for turning it off, FH2 came first with 50% and TF14 and OF17 came next with 11% each. The most preferred gestures to increase the temperature were OF22 with 25% and FH4 with 17%. To decrease the temperature, OF22 came first with a usage of 28%, followed by FH2, FH4 and FH6 with 14% each. The most preferred gesture to activate and deactivate the horizontal wind direction was FH7, with 42% and 31% usage, respectively, while the second most preferred gestured for the same two tasks was FH6. Participants mostly preferred FH8 and then FH4 to activate and deactivate the vertical wind direction. FH6, OF22 and FH4 were the most employed gestures to speed up and slow down the fan. Changing the AC mode either to the previous or next mode was mostly achieved using FH7, FH3 and FH6 but in different orders.

4.4. Discussion of HomeG

4.4.1. Tasks and Gestures

Among the tested AC tasks, there were dichotomous tasks such as turn on/off, activate/deactivate and speed up/slow down. Similar top gesture forms emerged for the dichotomous tasks. For instance, full-hand dynamic pose (FH2) was the most performed gesture for turning on and off the AC, and horizontally activating and deactivating the wind direction shared the same most performed gesture—full-hand dynamic pose and move leftward/rightward (FH7).

In our study, similar operations, such as increasing the temperature and increasing the fan speed, sometimes had relatively high matching in gesture forms, especially the top ones. This result is aligned with the results of [19], who found that similar gestures are used with similar operations. This might be a reasonable reason for eliminating the fan speed in some studies, especially those that investigate the gesture in high cognitive load scenarios (e.g., [9,10]). However, similarity in their operations does not always mean they share the same gesture forms. For instance, the activation of the horizontal and vertical wind directions do not share the exact same gesture forms. Although the two tasks have the same operation (i.e., activation), they have different natures (moving the wind direction horizontally or vertically). This clearly indicates the influence of the direction that is associated with the operation when employing the command gesture.

4.4.2. Gestures in Crossing Environments

Compared to the gestures used to control an in-vehicle AC system (GestDrive), HomeG showed superior suitability for elicited gestures used to control home AC functions. However, due to the relatively large number of gesture types in our gesture taxonomy, the user might not memorise the command gestures when moving from home to vehicle, especially given that driving is sometimes associated with a high cognitive load. Hence, we suggest combining HomeG and GestDrive and applying some modifications to them, as described below.

4.4.3. Gesture Forms and Directions

HomeG includes full hand and one- and two-finger gestures in addition to three-finger gestures, which do not exist in GestDrive. However, according to [19], the gestures employed by one to three fingers should not be differentiated. Following this finding results in the possibility of some modification to HomeG to obtain a suitable gesture taxonomy for operating the AC in the home and vehicle. One-, two- and three-finger gestures can be combined in one criterion if the gesture forms are the same. For instance, one-finger and two-finger poses moving leftward/rightward can be unified to be any-finger poses with a move leftward/rightward.

Different gesture forms (i.e., static and dynamic) associated with same direction (rightward/leftward, upward/downward and forward/backward) existed in our study but not in GestDrive. This might be due to the free space the user has at home, which is not found in a vehicle. Flexibility exists insofar as the gesture form in other studies [26,29] in which the focus is on the direction regardless of what the gesture form is. This allowed us to unify some gestures and re-group them into fewer categories to facilitate controlling AC systems in the home and vehicle. Additionally, it disambiguated the effects of gestures based on the gesture form. Therefore, it was possible to combine gestures that share the same part of the hand and direction regardless of whether the gesture form is static or dynamic. For instance, the user can apply either a static or dynamic full-hand gesture and move the hand leftward/rightward to adapt the wind direction horizontally. Table 5 shows the final adapted gesture vocabulary (CrossG) that possibly includes the most suitable gestures for controlling home and vehicle AC systems. The results and discussion related to this vocabulary are addressed below.

4.5. CrossG: Gesture Taxonomy for Gestures Crossing Environments

CrossG is a gesture taxonomy for gestures crossing environments, specifically between the home and vehicle. It consists of 13 gesture patterns which are derived from the modification process of HomeG. CrossG is more flexible than HomeG in identifying the same gestures but has fewer gesture types. This helps us to disambiguate the effects of gestures based on the form of the movement as well as the number of fingers used to employ the gestures. The distribution of the gesture patterns of CrossG is shown in Figure 8. Full-hand dynamic pose and turn (FH6), any-finger dynamic pose and turn (AF11) and full-hand dynamic pose (FH2) were the top three preferred gestures with 17.6%, 15.3% and 15.0% usage, respectively.

The number of gesture criteria assigned to each task following CrossG ranged from 7 to 10 criteria, with an average of 8.41 criteria for each task. The gesture distribution for each task based on CrossG is shown in Figure 9. The top three preferences for turning the AC on were FH2, AF12 and AF10, while FH2, AF8 and AF9 were the top gestures for turning the AC off. The most preferred gestures for adapting (i.e., increasing/decreasing) the temperature settings were AF11, AF12 and FH4. The gestures FH5 and FH4 were the first and second most preferred gestures for horizontally adapting (i.e., activating/deactivating) the wind direction. The most commonly employed gesture for activating the wind in a vertical direction was FH6 with 75%, while FH4 came next with 8%. For deactivating the vertical wind direction, FH6 came first with the relatively large proportion of 56%, FH1 came next with 11% and FH2 shared the third place with FH4 and AF9 at 8%. The participants mostly employed FH4, AF11 and FH6 when increasing/decreasing the fan speed. The last dichotomous tasks were adapting the AC mode. The most preferred gesture to shift the mode to the next was FH5 with 31%. FH5 was also used to shift the mode to the previous at 22% usage. Less likely, FH3 and FH4 were used to move to the next mode with 11%, and FH2 and FH4 were used to move to the previous mode with a usage of 14%.

4.6. Discussion of CrossG

Generally, CrossG had fewer gesture patterns than HomeG; the tasks were therefore associated with a narrower range of gestures. CrossG had a similar trend to HomeG in terms of the top gesture distributions for dichotomous tasks. Indeed, all dichotomous tasks had the same first top gesture. The similar tasks (i.e., increasing the temperature and fan speed and decreasing the temperature and fan speed) had relatively similar patterns for gesture distribution, particularly for the top gestures. The direction influenced the gesture distribution for the similar tasks (i.e., distributing the cold in a horizontal and a vertical manner and deactivating the direction of the wind in a horizontal and vertical manner).

5. General Discussion

5.1. Influence of Environment

Comparing the command gestures of our study with the gestures employed for controlling the in-vehicle AC reported in [9,10], the gestures used to control the home AC included the same gestures used to control the vehicle AC but with additional gestures and a greater variety of choices. In studies [9,10], the user of an in-vehicle AC system limited himself/herself by using only one hand. In contrast, the gestures in our study were derived from either one hand (left/right) or identically from both hands. Additionally, the gesture poses were derived using a full hand or one, two or three fingers. Furthermore, the user used more gesture forms when controlling the home AC than the in-vehicle AC. The reason for this might be that the purpose or objective of the main task is different in each environment. The driver aims to safely reach their destination. Thus, he/she concentrates on driving, looking at the driving field of view and keeping his/her hands on the steering wheel most of the time. He/she tries to ignore gaze and hand distractions as much he/she can in order to reach the desired goal of safely arriving at their destination. At home, however, the user has few constraints and is mostly looking for entertainment or relaxation. Hence, one gesture taxonomy cannot necessarily identify the gestures used to control the same device with the same functions in two different environments. More studies are required to understand the relationship between the environment factor and user choices when applying gestured-based systems.

5.2. Detailed or Undetailed Vocabularies

Detailed vocabularies, such as HomeG, consist of more gestures describing exactly the movement with an exact number of fingers/hands. In contrast, undetailed vocabularies, such as CrossG, are more general. They do not consider the exact number of gestures/hands. Hence, undetailed vocabularies have superior flexibility to detailed vocabularies. They can identify the same gesture set as a detailed gesture vocabulary but with fewer gesture patterns. Their flexibility comes from their consideration of the environment. They merge some patterns based on the part of hand that performs the gestures and the gesture form and direction. They deal with the dynamic and static poses that are generated by any finger similarly if they have the same direction. Hence, even in a relatively limited-space environment, the user is not very restricted in performing a gesture with a certain part of the hand in a certain form and movement in order to identify his/her gesture.

Furthermore, an undetailed gesture vocabulary makes it possible for gesture types from the most preferred gestures to be used when performing a specific task. For instance, by combining the gestures TF12, TF15, and OF19 from HomeG to produce the gesture AF13 in CrossG, the gesture AF13 becomes one of the top three gestures for horizontally activating the wind direction. Also, an undetailed gesture vocabulary increases the proportion of some top gestures and thus increases the probability of those gestures to be the preferred gesture for a certain task. For instance, following the home gesture taxonomy HomeG, the gesture F8 is the most preferred gesture for vertically activating the wind direction, with approximately 44% usage. This proportion increased to 75% for the same task when the gesture type F8 was combined with FH4 to produce the gesture FH6 in CrossG.

A detailed gesture vocabulary, however, has the superior ability to distinguish the differences between gestures. Unlike an undetailed vocabulary, a detailed vocabulary differentiates two gestures that have the same forms and pose but are generated by either two fingers or one finger. Therefore, it diminishes the ambiguity that arises from complex gestures that share some features. That makes a detailed vocabulary more suitable for systems used in a specific environment, such as in [9], where GestDrive was suitable for identifying the gestures for an in-vehicle AC system.

5.3. Influence of Direction

In some cases, the direction is explicitly (e.g., horizontally/vertically activate/deactivate the wind direction) or implicitly (e.g., increase/decrease the AC temperature and change the mode to the next/previous mode) associated with the AC function. In such cases, users tended to employ gesture poses with a direction or turn that simulates the direction associated with the AC function. For instance, when the directions “horizontal and vertical” are assigned to the same operation “activate”, the users employed the poses by moving their full hand or fingers leftward/rightward and upward/downward, respectively. Another example is that for the operation “change the mode” to the next or the previous option, the users applied moves (mostly leftward/rightward) to the command gestures. Nevertheless, the users preferred to perform static or dynamic poses without directions or turns.

5.4. Limitations and Future Steps

The task list in this study was provided in a particular order. The dichotomous tasks mostly followed each other, such as changing the AC mode to the next or previous mode. One idea for future studies is to test whether users generate different gestures when the list of tasks is randomly presented.

This study currently reports the elicited gestures for controlling a home AC system, comparing them with the gestures used in an in-vehicle AC system. The study also suggests some modifications for gesture taxonomy when classifying the gestures that can be used in the home and vehicle. Future studies could examine the suggested classification in both environments.

In terms of the sample variety, only men participated in the study. Although the study included right-, left- and mixed-handed participants, this study did not investigate the effect of handedness on the hand/hands that generated the gestures. Further studies with a broader sample could conduct further analysis regarding the impact of gender and handedness.

This study depended on an objective assessment method to explore the top gestures for the most common AC control commands. Hence, in future studies, the top gestures could be evaluated by additional methods, such as gestural agreement and the usability of user-elicited gestures.

6. Conclusions

With the increasing inclusion of technology in many aspects of everyday life, hand gestures have been used in smart homes and vehicles to control a wide range of various devices, including AC. Elicit studies provide a way to accurately define the preferred gestures for controlling a certain device by involving the actual end users in the gesture development lifecycle. The main contribution of this research is the proposal of a gesture vocabulary that efficiently identifies the gestures employed to control home and vehicle AC systems.

We conducted a user-elicited study with 36 participants who employed their preferred gestures to control 12 AC functions. The generated gestures for each AC function were analysed using a gesture vocabulary that was defined in previous studies when controlling an in-vehicle AC system. The analysis allowed us to define a new gesture vocabulary (HomeG) which contains 25 gesture patterns. The patterns are differentiated based on the parts of the hand (i.e., full hand, three figures, two fingers and one finger), pose form (static/dynamic) and direction. The results indicated that HomeG managed to identify all employed command gestures.

However, it would be useful to combine some patterns based on suggestions from previous studies to generate another undetailed gesture vocabulary. Different gesture forms (i.e., static and dynamic) associated with same direction (rightward/leftward, upward/downward, and forward/backward) were re-grouped together as they have the same influence on the gestures set, leading to a short, undetailed gesture vocabulary named CrossG. CrossG contains 13 gestures patterns which managed to identify all performed gestures in our study. This facilitates controlling AC systems when moving from the home to the vehicle and vice versa.

The detailed gesture vocabulary (HomeG) demonstrates superior differentiation ability over the undetailed gesture vocabulary (CrossG), making it more suitable for one specific environment. However, the undetailed gesture vocabulary has a shorter list of gestures and the ability to remove the ambiguity generated from the effects of the gesture form and direction. This might make CrossG more suitable across environments.

In the future, we plan to

• Conduct the same experiment with the same tasks but in a random order; thus, the dichotomous tasks will follow each other;
• Conduct the study with a broader sample, including a greater diversity in gender and a larger number of participants in order to conduct further analyses regarding the impact of gender and handedness;
• Test HomeG and CrossG in other environments, such as offices, and with different types and styles of AC, such as window and central AC;
• Evaluate the top gestures by additional methods, such as gestural agreement and the usability of user-elicited gestures.