Substitute Buttons: Exploring Tactile Perception of Physical Buttons for Use as Haptic Proxies

Abstract

Buttons are everywhere and are one of the most common interaction elements in both physical and digital interfaces. While virtual buttons offer versatility, enhancing them with realistic haptic feedback is challenging. Achieving this requires a comprehensive understanding of the tactile perception of physical buttons and their transferability to virtual counterparts. This research investigates tactile perception concerning button attributes such as shape, size, and roundness and their potential generalization across diverse button types. In our study, participants interacted with each of the 36 buttons in our search space and provided a response to which one they thought they were touching. The findings were used to establish six substitute buttons capable of effectively emulating tactile experiences across various buttons. In a second study, these substitute buttons were validated against virtual buttons in VR. Highlighting the potential use of the substitute buttons as haptic proxies for applications such as encountered-type haptics.

1. Introduction

Physical interfaces continue to hold significant relevance in user interface design, primarily due to the inherent absence of distinguishable buttons in entirely digital touch displays. For interactions devoid of visual guidance, as required by individuals with visual impairments, or scenarios necessitating user attention, such as controlling machinery, assembly processes, or wet lab applications, the tactile guidance of haptic feedback assumes paramount importance. The ability to locate and manipulate the correct button for a specific action without visual reliance translates into safer, swifter, and more straightforward interactions.

One domain that uses virtual user interfaces prominently is virtual reality (VR), which offers diverse, immersive visual environments that enable a broad range of experiences: storytelling, games, virtual twins, training, and more. However, to create fully immersive experiences, realistic haptic feedback is important [1,2]. One approach to providing haptic feedback is through the use of passive haptic feedback in the form of haptic proxies [3,4,5,6]. These haptic proxies are made with varying degrees of realism [7], serving as faithful replicas of virtual objects or approximations thereof.

Creating a fully immersive VR experience with haptic feedback for every virtual object comes with multiple challenges: mapping virtual objects to their physical counterparts [8], finding and creating the appropriate haptic proxies [7], and the allocation of sufficient physical space to accommodate these proxies. The scientific literature has proposed various strategies for mapping virtual objects to their physical counterparts. One approach involves manipulating the user’s physical movements by altering the virtual arm or the virtual environment, the so-called visuo-haptic illusions [9,10]. A complementary approach integrates passive haptic proxies with actuation mechanisms to reposition them, thereby generating encountered-type haptic interactions. Illustrative implementations encompass the use of robotics [11,12], such as robotic graphics [13,14] or quadcopters [15,16].

Our research contributes to the identification and construction of suitable haptic proxies and the minimization of the spatial requirements associated with these proxies. Striving to offer an exact haptic proxy for every virtual object is impractical and unfeasible: one can easily and instantaneously create a wide variety of virtual objects, while creating their exact physical counterparts is costly and requires time. Consequently, our efforts are centered on the development of haptic proxies capable of representing multiple virtual objects. This approach is viable due to the limited sensitivity of the haptic system and the dominant role of visual cues in human perception [17,18,19].

We introduce the concept of “substitute buttons”, comprising a set of six physical buttons designed to serve as versatile haptic proxies for a diverse range of virtual buttons. This contributes to the development of encountered-type haptic experiences and haptic devices capable of emulating an extended array of virtual buttons, all without necessitating a corresponding increase in the physical footprint that would be caused by providing haptic proxies for each virtual button.

This paper presents the following contributions:

• A comprehensive, in-depth analysis of the characteristics of physical buttons that play a significant role in tactile perception and button recognition.
• An exploratory study determining how users discriminate between different physical buttons in a blind test.
• The identification of a set of six substitute buttons that can imitate all physical buttons in our search space and potentially even more during VR interaction.
• A validation test to validate substitute buttons within a wide range of buttons.

Our findings are directly applicable in robotic graphics, in which a haptic experience is generated for virtual objects. More generally, non-visual haptic interactions and scenarios where interaction with physical buttons is crucial can benefit from our work.

2. Related Work

In this section, we discuss how our research is situated in the state of the art and provide the appropriate context on possible use cases for our results.

2.1. Characteristics of Buttons

Oulasvirta et al. [20] investigated the mechanics of pressing buttons, focusing on kinematics, dynamics, and user performance. One of the key aspects was to learn how users compare their expected and obtained sensations of button activation and then bridge this gap. Performance and user experience will likely decline if the gap is too big. Liao et al. [21] continued this work, providing methods to capture, edit, and simulate force-displacement, vibrations, and velocity-dependence characteristics. Our focus is not on the action of pressing buttons or the efficiency of button usage but complementary, as we look at the sensory perception of the shape and size of a button. The aim is to contribute to optimizing the user experience by using a limited set of physical buttons to represent a wide range of virtual buttons.

Others have investigated the tactile characteristics of buttons. Moore [22] aimed to identify physical button shapes with the best tactile discriminability. The buttons used in the study had protruding shapes, such as triangles or circles. Participants had to blindly identify each button by comparing it to its visual representation. The resulting confusion matrix was used to cluster the most distinctive buttons. Austin and Sleight [23] conducted a similar study on the accuracy of tactile discrimination of letters, numerals, and geometric forms. Four conditions were presented to participants (solid or outline figures, with or without finger movement), and the results showed no difference in discriminability based on gender, handedness, or individual fingers. Our study design is similar, but our objectives differ, as our study is focused on finding the least distinctive physical buttons instead of maximizing discriminability.

2.2. Tactile Spatial Resolution of the Fingertip

The Grating Orientation Task (GOT) is an established robust metric for gauging tactile spatial resolution and originates in the discipline of neuroscience [24]. This task, frequently referenced as the GOT task [25], requires participants to discern the orientation of a textured surface, specifically discriminating between horizontally and vertically oriented grooves through fingertip contact. The grating’s width is systematically manipulated, gradually decreasing until participants can no longer consistently distinguish the orientation with a threshold of 75 % accuracy. The human fingertip can differentiate orientations in gratings as narrow as one mm [26,27].

Tactile spatial resolution generally demonstrates an advantage in female participants, which can be attributed to differences in finger size. On average, women possess smaller fingers, which have been linked to a greater density of Merkel cells, resulting in a more finely detailed perception of tactile stimuli [28].

In contrast, physical buttons commonly integrated into user interfaces predominantly feature smooth surfaces, with their primary distinguishing feature residing in the button’s contour itself. Notably, these buttons lack the textured gratings or distinct orientations essential for recognition through touch, as exemplified by the grating orientation discrimination tasks. Consequently, it is reasonable to assert that the accuracy metrics reported in the aforementioned studies may not directly apply to our specific use case. Therefore, we contend that the tactile spatial resolution for buttons is likely to be coarser than the previously established threshold of one mm. Moreover, in light of the documented correlation between fingertip area and tactile spatial resolution, we intend to measure fingertip area to assess if this relationship holds true within the context of our study.

2.3. Passive Haptic Feedback

Passive haptic feedback is a type of haptic feedback that relies on physical objects with low complexity. As demonstrated by Shapira et al. [29], physical objects such as toys can be utilized as proxies for virtual objects in a VR setting. These proxies can be adapted to represent different virtual objects; e.g., a block can represent a house, binoculars, or a replica of the block. Exact replicas of virtual objects provide high levels of fidelity but are limited in their flexibility as a single prop can only correspond to a single or limited set of virtual items, and changes during runtime are hard to handle.

The primary question is the extent to which the fidelity of the props influences VR realism. Simeone et al. [7] study the impact of mismatches on the user experience in their exploration of substitutional reality. Every physical object surrounding the user is paired, with some degree of discrepancy, with a visual counterpart. They conclude that a more significant mismatch can increase the range of environments that can substitute for the physical environment (the physical proxy can be used in more situations). Still, they can also become an obstacle to the interaction and the experience’s plausibility.

We identify a limited set of physical buttons that can serve as proxies for a much wider variety of virtual buttons. Our study aims to determine the precise boundaries within which a physical button can effectively act as a proxy for multiple buttons without affecting the user’s perception. In contrast to previous work, we focus on buttons instead of generalizing this for a wide range of physical form factors. This allows us to gain a deep insight into how we can obtain a consistent combination of a physical and visual button without affecting the user perception, thus optimizing the immersive sensation for these elementary and ubiquitous interactors.

2.4. Visual Illusion Techniques

The human perception system is dominated by the visual system [17,18,19], making it possible to alter the user’s perception, particularly in VR, where we have control over all the visual information that is presented to the user. This idea has generated a significant body of research, encompassing a range of solutions that address various aspects of human perception in VR. Feick et al. [30] showed, for instance, that it is possible to reuse a single physical slider for multiple virtual sliders through the use of visual–haptic illusions such as linear translation and linear stretching. Zenner and Krüger [31] investigated the order of magnitude of hand redirection that can be applied without users noticing. Similarly, Feick et al. [32] study the impact of different grasping types, movement trajectories, and object masses on the discrepancy introduced by visuo–haptic illusions. VR Grabbers [33], a controller similar to training chopsticks, combines the passive haptic feedback from closing the chopsticks with a visual representation of grabbing tools, and the pseudo-haptic technique was found to increase task performance. Our work relies similarly on these techniques, using physical buttons as haptic proxies for various virtual buttons.

2.5. Encountered-Type Haptics

Encountered-type haptics, or robotic graphics [13,34], are actuation devices that provide haptic feedback to the user on demand at the point of interaction. Gruenbaum et al. [14] created a robotic system representing a virtual control panel by providing the appropriate input device at the right time. A similar approach by Latham [35] focuses on airplane cockpits. Snake Charmer [11] extends a robotic arm to render haptic feedback, including textures, interactions, and temperature. In VRRobot [12], a robotic arm provides on-demand haptic sensations, while a motion platform allows the user to walk around in VR while staying stationary to remain within reach of the robotic arm. HapticPanel [36] uses a 2D motion platform instead of a robotic arm to provide a low-cost DIY approach.

Another approach is the use of quadcopters to implement encountered-type haptics. The main advantage of quadcopters is the theoretically unlimited interacting volume. Yamaguchi et al. [37] use a paper hung from the quadcopter to represent reaction force. Abdullah et al. [15] represent stiffness and weight with the quadcopter’s thrust for 1D interactions. Abtahi et al. [16] present a safe-to-touch quadcopter that enables three haptic interactions: dynamic positioning of passive haptics, texture mapping, and animating passive props.

In general, encountered-type haptics provide two types of passive haptic feedback as proxies: approximations or exact copies of the virtual objects. Our results expand the applications of encountered-type haptics by broadening the use cases for physical buttons as proxies.

3. Button Exploration Space

Our study begins by defining a representative set of commonly used physical buttons. Alexander et al. [38] characterize the physical aspects of over 1500 household push buttons and present a list of 20 unique properties. Button shapes vary, with 71.8% being rectangular, 15.9% circular, 9.5% elliptical, and 2.8% other. Surface area distribution correlates with the finger’s width, with rectangular buttons most common in the 10 mm to 18 mm range (keyboard buttons). In comparison, circular and elliptical buttons have the highest distribution in the 5 to 10 mm range.

MIL-STD-1472F [39] specifies the US Department of Defense design criteria standard for human engineering used to design and develop military systems. The standard states that push buttons operated by a bare fingertip should have a size between 10 and 25 mm and a minimum size of 19 mm for a gloved fingertip. Buttons sold on a popular online store for electronic components (Newark https://www.newark.com/c/switches-relays/switches, accessed on 15 June 2022) come in similar sizes. These buttons would be used in practical user interfaces and potentially useful in encountered-type haptics applications. Indexing the (push) buttons sold for electronic components finds a total of 6319 different buttons, with a minimum size of 2 mm and a maximum size of 38 mm. The most common size is between 6 and 22 mm.

Based on these sources, we identified that the most common physical buttons range between 5 and 25 mm in height and width, and the most common shapes are square, rectangular, and round buttons. Therefore, we focus on these three main characteristics of a physical button: size, shape, and roundness.

The size impacts how much of the button fits underneath the fingertip of the user. The original search space consists of sizes between 5 and 25 mm. We reduced this to three sizes: 7, 11, and 15 mm. Size 15 is the maximum size of buttons because it is bigger than the surface area of the finger that comes into contact with the physical button. The average length of a male fingertip is 26 mm and of a female fingertip 24 mm [40], but only about 70% of the complete fingertip comes into contact with the button. We took 7 mm as the smallest size, as it allows for slightly smaller rectangular buttons. The selection of 11 mm as an intermediate size balances the other two dimensions, ensuring an array of sizes for evaluation and allowing for overlapping sizes when factoring in button shapes.

The shape is a big factor in the physical appearance of the button. We defined three shapes as the most common ones found in buttons: square, rectangular horizontal, and rectangular vertical. The rectangular horizontal and vertical buttons are identical but rotated 90 degrees. The rectangular buttons have two different lengths of sides. The largest length is the current size of the button. The smallest length of the side is the same size as one button size smaller. For example, the size 11 rectangular button is 11 by 7 mm. For the 7 mm button, the smallest side is 5 mm, which is the smallest size in our search space.

The roundness allows us to change the shape of buttons based on the roundness of the corners, thus enabling us to have slightly elliptical buttons as well. For this, we define four roundness factors: 0%, 33%, 66%, and 100%. Each rounding factor is applied uniformly to every corner of the button. We chose these four values as a step-based range between completely square and rounded corners. The roundness is calculated as half of the smallest size of the button multiplied by the roundness percentage. For example, the rectangular button of 15 by 11 mm with 66% roundness has a rounding of 3.66 mm (=(11/2) × 66%).

Based on these characteristics, we use three size, three shape, and four roundness options, resulting in 36 (3 × 3 × 4) physical buttons to test. These buttons are organized in a four-by-nine grid (Figure 1), organized as follows: the columns represent the roundness, starting in the first column with 0%. The rows represent the shape, starting with rectangular vertical, then rectangular horizontal, and square. This is repeated three times for each size group. The size is organized in groups, with the first three columns being size 7 mm, then 11 mm, and lastly, 15 mm.

4. Button Discrimination Test

The button discrimination test aims to identify the tactile features of buttons that can be accurately identified without visual cues. Participants receive sensory input exclusively through touch, without visual information about the button being pressed. The less a participant can differentiate between physical buttons, the more feasible it becomes to assign multiple virtual buttons to a single physical button. For this purpose, we perform a blind test using the previously defined button set, in which the participant can only feel the button without seeing it. Our study design was inspired by the work of Austin and Sleight [23] and Moore [22].

4.1. Study Design

This section describes the lab study design and the analysis of the results.

4.1.1. Apparatus

The central element of the experimental setup is a custom-made test platform (Figure 2a), housing all 36 (3D-printed) buttons arranged in a grid (Figure 1). The platform contains a button holder, as depicted in Figure 2a. Participants place their hand on the test platform and position their index finger above the designated testing aperture (as depicted in Figure 2b). The physical button being evaluated is placed beneath this aperture by the experimenter.

The holder incorporates all physical buttons and corresponding cutouts for each button. The position indicator on the platform and the numbered physical cutouts on the button holder facilitate easy placement of the correct button. The button tester has seven internal guide channels aligning with the physical guides on the button holder. The correct button placement is achieved by aligning the button holder with the correct column and positioning it to match the designated button in the correct row.

The study apparatus is fully enclosed in a cardboard box to eliminate any visual cues of the physical buttons (Figure 3a). An opening on top enables the experimenter to observe the test platform to ensure the accurate placement of physical buttons and monitor the participant’s finger movement (Figure 3b). A full-scale printout of all physical buttons is positioned on top of the box. The experimenter has a copy of the printout for reference during button placement.

4.1.2. Participants

A total of 30 individuals participated in the study, with an equal gender distribution. The sample was selected to provide a diverse range of finger sizes, as prior research has established a correlation between fingertip area and perceived tactile detail [28]. Participants needed to have healthy hands without factors affecting fingertip haptic sensitivity, such as calluses. We did not filter the participants on their occupation as we believe button pressing is ubiquitous. The average age of participants was 34, ranging from 21 to 62. Two participants were left-handed, and all others were right-handed. Every participant used their dominant hand to interact with the buttons, which did not affect the results. When pressing a physical button, the total area of the fingertip does not come into contact with the top of the button but only with an ellipse in the middle of the fingertip. Therefore, we measured and calculated this area’s (width and height). Using our custom measurement apparatus (see Figure 4), the measurement is based on a picture taken from the participant’s index finger. It consists of a webcam and, above it, a Plexiglas plate with reference lines with a known spacing (2 mm) used in the surface area calculation. The average width was 13.0 mm, with a minimum of 11.1 and a maximum of 15.4 mm. The average height was 17.3 mm, with a minimum of 14.8 and a maximum of 19.9 mm. The average area was 177.7 mm², with a minimum of 129.5 and a maximum of 240.6 mm².

4.2. Study Procedure

The study procedure consists of a training phase and the main study phase. The training begins with instructing the participant on the right button pressing technique and hand placement. Each participant must press every button consistently and uniformly. Participants pressed each button once, maintaining finger contact for a maximum of five seconds without moving to assess its characteristics. The opening on top of the box is utilized to monitor the participant’s compliance, while a timer indicates the end of the five-second duration.

The second part of the training explains the organization of the reference sheet with all physical buttons (same as is annotated in Figure 1) and the study procedure. The training concludes with a practice round of three buttons. These are buttons with 66% rounding, one of each size, and with a random selection of shapes, yet consistent across the three buttons for each participant. For example, a participant might test button 35 , button 23 , and button 11 . For every button, the participant is asked to verbally state the number of the physical button they believed they had pressed. Participants were only allowed to answer one number per physical button and thus had to make a choice.

During the study, participants were tasked with pressing all 36 physical buttons in random order. They were encouraged to think out loud and explain their thought process. Some reported considering multiple options before deciding on a specific button. After the study, a short informal interview was conducted. Four questions were asked: how well they thought they did in recognizing the roundness, shape, size, and number of correct answers. Participants provided scores from 1 to 10 for each question and their reasoning. We used this information to gain insight into the participants’ decision-making process. Finally, a picture of each participant’s index fingertip was taken.

Throughout the study, participants did not receive feedback on their button selections or accuracy. The participants could not reliably determine the physical button they interacted with and thus could not determine the next physical button based on the movement of the button holder in the study apparatus.

4.3. Study Results

The first outcome of our study was the accuracy with which participants identified the physical buttons they pressed, as measured by the number of buttons correctly answered. It ranges from 3 to 12 correct answers, with an average score of 7.26 and a standard deviation of 2.56. The high variability, with the best-performing participant correctly identifying only a third of the buttons, supports our hypothesis that touch-based button recognition is challenging.

Further analysis of participants’ answers, the think-aloud data, and post-study interviews revealed that accurately recognizing the physical buttons was challenging due to the relatively small differences between the buttons and the difficulty in estimating the correct level of roundness. Participants tended to guess buttons with 33% and 66% roundness from the middle column, as shown in Figure 5. This suggests that participants guessed values in the middle in cases of uncertainty.

No statistically significant correlation was identified between participants’ fingertip area (Pearson’s r(28) = −0.03, p < 0.44), gender (Pearson’s r(28) = −0.29, p < 0.06), or age (Pearson’s r(28) = −0.02, p < 0.45) and the number of correct answers.

5. Determining the Substitute Buttons

We employ a comprehensive analytical procedure to establish a set of substitute buttons. First, we portray the physical button and the corresponding responses from participants through a confusion matrix. This matrix offers valuable insights into the patterns and tendencies observed in the participant responses. Subsequently, we apply hierarchical clustering techniques, utilizing the confusion matrix as the fundamental data source. This clustering methodology draws inspiration from the method outlined in Moore’s work [22].

5.1. Confusion Matrix Exploration

The confusion matrix is a 36 by 36 matrix, with each row representing a physical button and the answers given for each physical button (the columns). Each cell of the confusion matrix represents the number of responses for that specific combination of physical and answer buttons (e.g., for physical button 5 , button 6 was chosen nine times). The complete confusion matrix is presented in Figure 6, with a visualization representing the button size, shape, and roundness of 0 and 100% next to the column and rows. The three highlighted squares correspond to the three size groups, with sizes 15, 11, and 7, respectively. Each group of four rows or columns represents a different shape, and each individual row represents a specific rounding value. The confusion matrix offers valuable visual insights into the data and the recognizability of the buttons.

Clustering is observed around the distinct shape groups in a 4 by 4 grid grouped by roundness. The cluster formation is evident in button groups 17 –20 and answers 17 –20 , representing the four rectangular horizontal buttons with roundness values of 0, 33, 66, and 100%. These results suggest that the shape is highly distinguishable, particularly for rectangular horizontal buttons, while the roundness is more challenging to discern. A similar pattern is observed in button groups 21 –24 , which consist of four rectangular vertical buttons with size 11. Furthermore, an analysis of each size group indicates that the rectangular horizontal button is the most recognizable among all shapes for every size.

The distribution of answers reveals the impact of size on recognition accuracy. For buttons with size 15, most answers are correct regarding size, while some are answers for the rectangular horizontal buttons with size 11 (buttons 17 –19 ). This trend is also evident for buttons with size 7, where most answers outside their size group are for buttons 17 –19 . This can be attributed to the fact that buttons with size 15 share the same width, and buttons with size 7 share the same height. For buttons with size 11, the distribution of answers is more dispersed among sizes, with most answers being for size 11, followed by size 15. This is especially pronounced for square buttons with size 11 (buttons 13 –15 ), where many answers are given to the rectangular horizontal buttons with size 15 (buttons 5 –7 ). This can be explained by the fact that buttons 13 –15 fit inside buttons 5 –7 , with the only difference being the length of the horizontal sides. Hence, when participants are uncertain about the size or find it difficult to estimate the correct size based on the left and right edges, they tend to choose buttons 5 –7 .

An analysis of the columns corresponding to buttons with 100% roundness reveals a low frequency of correct answers. One notable exception is button 16 , which is readily recognized as a round button with either 66% (button 15 ) or 100% roundness. This can be attributed to the size of 11 mm, which falls within the average fingertip width (13.0 mm) and height (17.3 mm), making it easily distinguishable as round. Button 24 represents another exceptional case with no accurate answers. The most frequently guessed answer was button 23 , suggesting participants underestimated the roundness. The frequency of correct answers is higher for buttons with size 7 and 100% roundness. This can be attributed to the small size of the buttons, which, in combination with the rounding, makes it difficult to interpret the shape and roundness accurately, but participants do interpret some of these buttons as having 100% roundness.

5.2. Clustering Approach

To determine the substitute buttons, we apply a hierarchical clustering approach. The clusters obtained from the clustering analysis are then utilized to identify the substitute buttons that can effectively represent the entire set of 36 buttons.

To perform the clustering analysis, we calculate the Euclidean distances between the physical buttons (represented as rows) based on the values in the confusion matrix. For instance, when examining the rows corresponding to physical buttons 5 and 6, they have similar values for each column, resulting in a small distance between them (their perceived similarity was high). Then, sklearn’s Single Linkage Clustering algorithm [41,42] was applied. This algorithm initiates with each button as an independent cluster and progressively merges the two closest clusters until only a single cluster remains. The resultant hierarchical structure is visualized as a dendrogram, as illustrated in Figure 7. To extract meaningful clusters, we applied a threshold distance of 9.3. This threshold identified six distinct representative clusters, with an additional four buttons necessitating manual assignment to one of the clusters.

We aimed for a maximum of nine clusters, resulting in one cluster for every shape–size combination, but a lower number of clusters is more desirable. Therefore, the threshold distance of 9.3 is picked, as it generates six distinct clusters. This results in a six-fold reduction in the buttons in the original search space.

The separate buttons, 12 , 15 , 16 , and 20 (highlighted in blue in the dendrogram), were assigned to clusters through manual inspection of the confusion matrix and their distances to the existing clusters:

• Button 20 . The nearest neighbor is button 19 of cluster 3. Further examination of the confusion matrix reveals that button 19 is the most frequently classified button for button 20. The size and shape of button 20 also resemble the buttons in cluster 3.
• Button 12 . The nearest neighbor is button 9 of cluster 1. Further examination of the confusion matrix shows that the most frequently classified buttons for button 12 belong to cluster 1.
• Button 15 . The nearest neighbor is 14 of cluster 2. The confusion matrix also shows similarities between the rows of button 14 and button 15. Button 6 , also belonging to cluster 2, was the second most commonly answered button, with button 15.
• Button 16 . The nearest neighbor is button 13 of cluster 2. The confusion matrix shows that button 15 is also the most answered button for button 16, which is also a part of cluster 2.

Finally, each cluster’s most central button was determined by selecting the button with the lowest total distance to the other buttons within the cluster, as this button best represents the other buttons in its cluster. The center buttons for clusters 1 to 6, which serve as a representative substitute for their cluster, were identified as buttons 1 , 8 , 19 , 23 , 30 , and 36 , respectively. The results are illustrated in Figure 8, with each cluster represented by a different color and the center button highlighted.

6. Validation of the Substitute Buttons

Based on our first test, we found six candidate substitute buttons. With our second study we want to validate how well these substitute buttons can represent virtual buttons inside a VR environment. In this study, the visual information does not necessarily match the physical information provided by the physical buttons, allowing us to also test if the added visual information changes the impression of the tactile sensations of the physical buttons.

6.1. Study Design

Our study design closely resembles our first study: participants interact with a physical button through our test platform and answer questions about the physical button. The main difference in this study is that the participants are wearing a VR HMD (Varjo Aero https://varjo.com/products/aero/, accessed on 10 August 2023) and are shown virtual buttons (Figure 9b).

6.1.1. Apparatus

The apparatus is an updated version of the test platform from the button discrimination test. It houses nine 3D-printed buttons that move along only one axis, as shown in Figure 9a. These buttons consist of the six substitute buttons and three control group buttons (12 , 13 , 25 ). For the control group buttons, we selected one button from each size group, ensuring maximal dissimilarity between each control button and its corresponding substitute button of equivalent size. Participants place their dominant hand on the platform and press the button through the aperture. The buttons are connected through an Arduino Uno with the VR environment made in Unity to signal the physical button press to the virtual environment. The experimenter is again responsible for positioning the correct button underneath the aperture and recording the response from the participant.

6.1.2. Participants

Sixteen right-handed participants (15 male, 1 female), aged 20 to 28, participated, with a mean age of 21.5. As our first study showed no correlation between finger size, age, or gender and button recognition, we did not explicitly balance our participants for finger size.

6.2. Study Procedure

The study consists of a training phase and the main study. During the training, participants are guided on hand placement and button pressing. Each participant may only press the button once and is not allowed to maintain contact with the button. This approach is chosen as it mimics real interaction with a button. After every button press, the participant is asked to rate the degree of similarity between the physical button they pressed and the virtual button they saw. We use an ordinal rating scale: 1 = Same, 2 = Unsure, 3 = Different. During the training, the participant will test the three control group buttons, which are always presented together with their matching virtual button.

During the study, the participants tested every substitute button with all the buttons inside its cluster and their neighboring clusters (96 tests). For example, substitute button 19 is tested with all the buttons from cluster 3 (own cluster) and clusters 2 and 4 (neighboring clusters). The three control group buttons are also tested once, resulting in 99 buttons tested by each participant. Participants received no feedback during the study on their answers to minimize the learning effect.

6.3. Results

Our analysis of this study is based on the similarity of each substitute button compared to the virtual buttons of its own cluster and the neighboring clusters. The similarity is defined in the three ordinal categories: same, unsure, and different. Figure 10 visually represents the data, visually showing the distribution of the answers given over the virtual buttons. Every Substitute Button is represented in its own row with three sub-rows for the three possible answers. Every column represents a virtual button that is shown in the VR environment. The numbers represent the number of times a specific answer is given for a specific combination of substitute button and virtual button. The color gradient maps the median to a specific color: dark blue for a median < 2; light blue for a median = 2; grey for a median = 2.5; and no color for a median = 3.

The visualization provides insight into the distribution of the answers. For example, Substitute Button 19 (green cluster) received mostly ‘same’ answers for the similarity inside its own cluster (buttons 17–20). That same substitute button received mostly ‘different’ answers for the similarity compared against cluster 4 (buttons 21–24). The reverse is true for substitute button 23 (orange cluster). It received a lot of ‘same’ answers for its own cluster and a lot of ‘different’ answers for its neighboring cluster.

Looking at the answers for the buttons of the control, we see that for all three buttons, not all participants found the buttons matching (see

Table 1. The distribution of answers over the three control group buttons, with every physical and virtual button being the same.

Control Group Button	12	13	25
Same	12	11	14
Unsure	2	4	0
Different	2	1	2
Median score	1	1	1

). Button 13 scored the worst out of the three, as only 11 participants indicated it was the same button. All buttons scored a median score of 1.

6.4. Analysis

We look at the median score for every combination to analyze how well every substitute button can represent the virtual buttons. The median scores indicate the distribution of answers given for each combination of physical and virtual buttons. Our goal is to find buttons that feel similar enough to the participants, so we are interested in the answers when participants indicated they were not sure if it was the same button and when they indicated it was the same button. Therefore, we put our detection threshold for feasible combinations at a median score of two.

Most substitute button/virtual button combinations within the same cluster scored a median score of two or lower. This shows that our clustering based on the button discrimination test results in appropriate substitute buttons for our original search space.

We also investigate the combinations of substitute and virtual buttons tested outside the corresponding substitute cluster. These represent substitute buttons that are recognized as the virtual button outside the original cluster; these can also be seen in Figure 10. Substitute button 1 can represent all virtual buttons from 1 to 16 besides buttons 12 and 16. Substitute button 8 can represent buttons 1, 2, 3, 10, 20. Substitute button 19 can represent buttons 5 and 7, buttons of the same shape but one size larger. Substitute button 23 can represent buttons 34 and 35, buttons of the same shape but one size smaller. Substitute button 30 can represent buttons 26 and 27.

These results show that some substitute buttons might also work for a broader range of virtual buttons outside of their original cluster. More specifically, substitute button 1 , as it could represent all buttons of size 15 and even some of size 11. Similarly, for substitute button 36 , our results show that it could also represent all buttons of size 7.

The three control group buttons highlight that correctly interpreting the tactile sensation of the physical button is difficult, as these button combinations are not recognized as the same by all the participants (see Table 1). Some participants report these combinations as unsure or even different.

We found three button combinations (see

Table 2. The three combinations of substitute/virtual button for which the median score is higher than 2, indicating that the combination is most often recognized as different buttons.

Substitute Button	Virtual Button	Corresponding Cluster	Median Score
1	12	1	2.5
8	16	2	3
30	32	6	2.5

) for which the substitute button is not an appropriate match for the virtual button. Combined with Figure 10, we can see the distribution of the answers for these combinations.

• Button 12. Half of the participants recognize a difference between substitute button 1 and virtual button 12 , while the other half believe they are the same or are unsure about the difference.
• Button 16. For this combination, 11 out of 16 participants recognized a difference between substitute button 8 and button 16 . But, if we look at the combination of substitute button 1 and button 16 , we can again see a 50–50 distribution.
• Button 32. The combination of substitute button 30 and button 32 is again a 50–50 distribution. Still, here we can see that substitute button 36 and button 32 have only seven different ratings, indicating that the difference in recognition between these two combinations is low.

Because the control group did not score perfect recognition and some button combinations scored better outside their cluster, we propose the following solutions for the three outlier button combinations: 1–12, 8–16, 30–32. Virtual buttons 12 and 32 are both on a median of 2.5 for their respective substitute button but might still work as a substitute button/virtual button combination. Half of the participants found the matching passable; the control group showed difficulty in recognizing the tactile sensations, and we tested in controlled circumstances, while the actual use case might be different. Substitute button 8 in combination with button 16 is not a good match, but in combination with substitute button 1 , it also scores a median of 2.5. Thus, for the same reasons as buttons 12 and 32 , we hypothesize that button 16 is a better fit with substitute button 1. Our findings show that there is only one button that could move to another cluster: button 16 has a stronger association with substitute button 1 than 8 andthus might move from cluster 2 to cluster 1.

7. Applications

To situate our research within the existing literature and haptic feedback devices, we describe several use cases for our results. Secondly, a field study where we test our results within a virtual reality game context is conducted and explained.

7.1. Applications within the Literature

Applying the results of our study, the six substitute buttons enable various haptic devices and applications to increase the number of virtual buttons they can represent without increasing the physical space required for them. Depending on the number of virtual buttons that need representing, at most, the six substitute buttons are required.

The first type of haptic user interface for which our results can be applied are encountered haptic devices such as Snake Charmer [11], HapticPanel [36], VRRobot [12], and robotic graphics applications [13,14] in general.

A second type of haptic user interface is handheld haptic devices such as the Haptic Revolver [43]. They show two example cases where the actuated wheel of the device is fitted with buttons: one set of active buttons and one set of passive buttons. Our six substitute buttons could perfectly fit on the wheel, thus enabling full haptic feedback on all virtual buttons in our search space from using only one haptic device.

A third type of haptic user interface that benefits from our work is those that use haptic retargeting [44]. The work of Matthews et al. [45,46] shows the possibilities of haptic retargeting for physical–virtual interfaces. We build on this work by providing guidelines for reusing a single physical button for multiple virtual buttons. Combined with haptic retargeting, it allows a physical button to represent multiple virtual buttons in different positions in the virtual interface.

Our work expands the literature on the use of passive haptic proxies by increasing the reusability of the haptic proxies for more virtual objects, specifically virtual buttons. In combination with the work of Feick et al. [30] on reusing physical sliders and linear movement, and DynaKnob [47], a shape-changing knob, we envision complete virtual interfaces represented by a compact set of haptic proxies for each interaction type, i.e., sliders, buttons, knobs.

Lastly, our research contributes to the literature on button interaction by examining how physical button characteristics are perceived with and without visual information. Our findings demonstrate that specific physical characteristics remain distinguishable even without visual information. Linking these specific buttons to different actions enables the creation of interfaces optimized for interaction without looking at them. This work extends the research of Oulasvirta et al. [20] and Liao et al. [21], who focused on the characteristics of internal button mechanisms while pressing them and simulating these internal mechanisms. Our work enables the reuse of the physical characteristics of the buttons. Combining these allows us to theorize a physical interface consisting of the six substitute buttons, with each a dynamic internal mechanism, creating a dynamic physical interface that can represent a broad range of physical buttons in physical and internal characteristics.

7.2. Informal Field Test

We executed an informal field test during a local popular science festival (Nerdland Festival—https://www.nerdlandfestival.be/nl/herbeleef, accessed on 19 February 2024). This test aimed to evaluate the practical usability of our substitute buttons in a more realistic interactive scenario. Participants played a simple VR game using physical buttons, diverting their focus from the pressed button. In contrast to the previous controlled experiments, participants engaged in a simplified VR game that required them to interact with the substitute buttons (see Figure 11a). The game objective was to synchronize pressing the physical button with the overlap of the virtual button, inspired by Guitar Hero’s (https://en.wikipedia.org/wiki/Guitar_Hero, accessed on 11 August 2023) gameplay; see Figure 11b.

Participants had the same six physical buttons (see Figure 11a) in front of them, with the virtual button changing between rounds. Following each round, participants were queried regarding any distinctions they observed between the physical and virtual buttons and whether these distinctions influenced their overall enjoyment of the game.

Our informal field test tested each of the 6 substitute buttons in conjunction with 12 distinct virtual buttons. These virtual buttons included those within the same cluster and a subset of neighboring clusters, dependent on the cluster sizes. Each combination involving a substitute button was iterated three times in a round-robin fashion. After completing the full round, the physical buttons were transitioned to the next substitute button. Each participant participated in three game rounds, engaging with three unique combinations of substitute and virtual buttons. The test enlisted the participation of 64 individuals, with demographic information such as age and gender not recorded to streamline the procedure, as our first study showed no correlation between age, finger size, and gender and button recognizability.

Given the test’s integration into the science festival, we opted for brevity, aiming to maintain a duration of a maximum of 5 min, including the explanation. The number of participants combined with the number of tests yielded an average of four responses per combination. Overall, the setup provided a positive experience, as most participants did not perceive the mismatch between physical and virtual buttons. Only where the difference between physical and virtual buttons was outside of the similar clusters, the mismatch was noticed but not detrimental to the game experience.

8. Discussion

We conducted two studies to determine the discriminability of buttons in relation to the shape, size, and roundness of the corners to find a subset of buttons that can represent all the buttons in the original search space. We tested in two different circumstances: without any visual information and with visual information provided through a VR headset. We started with testing without any visual information, as this forced the participants to judge the buttons based solely on the haptic characteristics and without any visual information influencing their decision. This results in baseline data created in the most challenging circumstances. The research has shown that visual information can influence haptic sensations [17,18,19], making these results more generally applicable.

For the second study, we added the visual information of a virtual button through the VR headset, potentially influencing the decision of the physical characteristics of the physical–virtual button combination. For the main part, we see that our results from the first study still apply in this case. Substitute buttons 1 and 36 can represent a larger set of virtual buttons than their original cluster. But, there is a limit to the mismatch between the physical button and virtual buttons that the visual information can overcome. For example, substitute buttons 19 and 23 clearly show that tactile sensations are more important when the mismatch between physical and virtual information is too big. These buttons were easy to identify in the first study, which is still the case for the second study, even with conflicting visual information, showing us that there are limits to the mismatch in haptic versus visual information that can be overcome by visual information only. If the mismatch is too large, the haptic information is the information channel that dominates the users’ perception. For substitute buttons 19 and 23 , we theorize that this is because of the size of 11 mm of these buttons. This size fits nicely inside the average fingertip area and thus provides a very clear tactile sensation on the shape of this button.

The participant’s finger was positioned flat on the testing apparatus (Figure 2b). The horizontal positioning of the experimental setup was chosen to maximize the contact area between the fingertip and the button, facilitating more informed choices. In the case of a vertical orientation, this contact area is likely smaller, as it would be mostly the tip at the front of the finger, thus decreasing the “amount” of haptic sensing. Therefore, we believe our results also apply to the vertical orientation. However, further testing is needed to validate these assumptions.

Our findings allow us to make assumptions about recognizing buttons beyond the original search space. The two most crucial factors to consider are the shape and size of the buttons. The size is the most easily recognizable attribute, with the shape following closely behind, albeit influenced by size. When searching for a substitute button for a virtual button not included in our study, it is advisable first to identify the closest size group and seek the cluster that best fits that shape. In cases where the difference between size groups is equal, such as a button of size 13 mm, it is recommended to round up, as most misinterpretations of size were overestimations.

Buttons larger than 15 mm can be effectively approximated using button 1 from cluster 1. This is because button 1 is larger than the average fingertip area that makes contact with the button. Any variations in size would not be perceptible as long as the button interaction takes place in the center. On the other hand, buttons with a size smaller than 7 mm can be represented by button 36 , which corresponds to cluster 5. Given their small size, these buttons are likely to pose difficulties in differentiation, and their usage during typical interactions is relatively rare.

The degree of roundness in the button was found to be of secondary importance. Our participants in the first study encountered difficulties in accurately determining the roundness and instead made educated guesses. Hence, when seeking a substitute button, we recommended prioritizing finding a match in terms of shape and size and only considering the roundness as a factor when it lies in the extremes: close to 0% or 100% roundness. Our validation study confirms that the combinations with 0% and 100% roundness result in higher uncertainty about the button matching.

9. Conclusions

In this paper, we introduced the concept of substitute buttons, a set of carefully selected physical buttons (, , , , , ) that represent a wide range of virtual buttons. These substitute buttons were identified in a perception study utilizing a blind identification test. We used a hierarchical clustering approach based on the confusion matrix from the data we gathered to classify groups of physical buttons that a single substitute button can represent. We conducted a second study adding a visual channel using virtual reality (VR) to validate the effectiveness of these substitute buttons in various physical–virtual button combinations. Our findings highlight the challenges of identifying physical buttons solely based on tactile sensations and enabling the reuse of substitute buttons to represent multiple VR buttons while preserving appropriate haptic feedback. This work opens new avenues for enhancing haptic feedback in VR applications by reducing the number of haptic proxies required. For instance, it can be applied to diverse haptic devices such as encountered-type haptics [11,36], hand-held haptics [43], or interfaces developed through haptic retargeting [45]. Doing so facilitates the development of a broader range of VR applications where appropriate haptic interaction with buttons is important.