Multimodal Technologies and Interaction

18 pages, 1160 KiB

Open AccessArticle

Developing Dynamic Audio Navigation UIs to Pinpoint Elements in Tactile Graphics

by Gaspar Ramôa, Vincent Schmidt and Peter König

Multimodal Technol. Interact. 2022, 6(12), 113; https://doi.org/10.3390/mti6120113 - 18 Dec 2022

Cited by 4 | Viewed by 1793

Access to complex graphical information is essential when connecting blind and visually impaired (BVI) people with the world. Tactile graphics readers enable access to graphical data through audio-tactile user interfaces (UIs), but these have yet to mature. A challenging task for blind people [...] Read more.

Access to complex graphical information is essential when connecting blind and visually impaired (BVI) people with the world. Tactile graphics readers enable access to graphical data through audio-tactile user interfaces (UIs), but these have yet to mature. A challenging task for blind people is locating specific elements–areas in detailed tactile graphics. To this end, we developed three audio navigation UIs that dynamically guide the user’s hand to a specific position using audio feedback. One is based on submarine sonar sounds, another relies on the target’s coordinate plan x and y-axis, and the last uses direct voice instructions. The UIs were implemented in the Tactonom Reader device, a new tactile graphic reader that enhances swell paper graphics with pinpointed audio explanations. To evaluate the effectiveness of the three different dynamic navigation UIs, we conducted a within-subject usability test that involved 13 BVI participants. Beyond comparing the effectiveness of the different UIs, we observed and recorded the interaction of the visually impaired participants with the different navigation UI to further investigate their behavioral patterns during the interaction. We observed that user interfaces that required the user to move their hand in a straight direction were more likely to provoke frustration and were often perceived as challenging for blind and visually impaired people. The analysis revealed that the voice-based navigation UI guides the participant the fastest to the target and does not require prior training. This suggests that a voice-based navigation strategy is a promising approach for designing an accessible user interface for the blind. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives)

► Show Figures

Figure 1

16 pages, 1354 KiB

Open AccessArticle

A Machine Learning Approach to Identify the Preferred Representational System of a Person

by Mohammad Hossein Amirhosseini and Julie Wall

Multimodal Technol. Interact. 2022, 6(12), 112; https://doi.org/10.3390/mti6120112 - 17 Dec 2022

Viewed by 1635

Abstract

Whenever people think about something or engage in activities, internal mental processes will be engaged. These processes consist of sensory representations, such as visual, auditory, and kinesthetic, which are constantly being used, and they can have an impact on a person’s performance. Each [...] Read more.

Whenever people think about something or engage in activities, internal mental processes will be engaged. These processes consist of sensory representations, such as visual, auditory, and kinesthetic, which are constantly being used, and they can have an impact on a person’s performance. Each person has a preferred representational system they use most when speaking, learning, or communicating, and identifying it can explain a large part of their exhibited behaviours and characteristics. This paper proposes a machine learning-based automated approach to identify the preferred representational system of a person that is used unconsciously. A novel methodology has been used to create a specific labelled conversational dataset, four different machine learning models (support vector machine, logistic regression, random forest, and k-nearest neighbour) have been implemented, and the performance of these models has been evaluated and compared. The results show that the support vector machine model has the best performance for identifying a person’s preferred representational system, as it has a better mean accuracy score compared to the other approaches after the performance of 10-fold cross-validation. The automated model proposed here can assist Neuro Linguistic Programming practitioners and psychologists to have a better understanding of their clients’ behavioural patterns and the relevant cognitive processes. It can also be used by people and organisations in order to achieve their goals in personal development and management. The two main knowledge contributions in this paper are the creation of the first labelled dataset for representational systems, which is now publicly available, and the use of machine learning techniques for the first time to identify a person’s preferred representational system in an automated way. Full article

(This article belongs to the Special Issue Feature Papers in Multimodal Technologies and Interaction—Edition 2023)

► Show Figures

Figure 1

21 pages, 7701 KiB

Open AccessArticle

patchIT: A Multipurpose Patch Creation Tool for Image Processing Applications

by Anastasios L. Kesidis, Vassilios Krassanakis, Loukas-Moysis Misthos and Nikolaos Merlemis

Multimodal Technol. Interact. 2022, 6(12), 111; https://doi.org/10.3390/mti6120111 - 14 Dec 2022

Viewed by 4771

Abstract

Patch-based approaches in image processing are often preferable to working with the entire image. They provide an alternative representation of the image as a set of partial local sub-images (patches) which is a vital preprocessing step in many image processing applications. In this [...] Read more.

Patch-based approaches in image processing are often preferable to working with the entire image. They provide an alternative representation of the image as a set of partial local sub-images (patches) which is a vital preprocessing step in many image processing applications. In this paper, a new software tool called patchIT is presented, providing an integrated framework suitable for the systematic and automatized extraction of patches from images based on user-defined geometrical and spatial criteria. Patches can be extracted in both a sliding and random manner and can be exported either as images, MATLAB .mat files, or raw text files. The proposed tool offers further functionality, including masking operations that act as spatial filters, identifying candidate patch areas, as well as geometric transformations by applying patch value indexing. It also efficiently handles issues that arise in large-scale patch processing scenarios in terms of memory and time requirements. In addition, a use case in cartographic research is presented that utilizes patchIT for map evaluation purposes based on a visual heterogeneity indicator. The tool supports all common image file formats and efficiently processes bitonal, grayscale, color, and multispectral images. PatchIT is freely available to the scientific community under the third version of GNU General Public License (GPL v3) on the GitHub platform. Full article

(This article belongs to the Special Issue Feature Papers in Multimodal Technologies and Interaction—Edition 2023)

► Show Figures

Figure 1

20 pages, 2572 KiB

Open AccessArticle

Virtual Reality in Health Science Education: Professors’ Perceptions

by Álvaro Antón-Sancho, Pablo Fernández-Arias and Diego Vergara

Multimodal Technol. Interact. 2022, 6(12), 110; https://doi.org/10.3390/mti6120110 - 14 Dec 2022

Cited by 11 | Viewed by 3555

Abstract

Virtual reality (VR) is a simulated experience in a three-dimensional (3D) computer-simulated world. Recent advances in technology position VR as a multipurpose technology in the healthcare sector and as a critical component in achieving Health 4.0. In this article, descriptive and correlationally quantitative [...] Read more.

Virtual reality (VR) is a simulated experience in a three-dimensional (3D) computer-simulated world. Recent advances in technology position VR as a multipurpose technology in the healthcare sector and as a critical component in achieving Health 4.0. In this article, descriptive and correlationally quantitative research is carried out on the assessments made by Latin American health sciences university professors on the didactic use of virtual reality technologies. The main objective was to analyze the differences in the perceptions expressed by the public or private tenure of the universities where the professors teach. In addition, gender and age gaps were identified in the assessments obtained from each of the types of universities. The results reveal that Latin American health science professors at private universities have a higher selfconcept of their digital skills for the use of virtual reality in the lectures. This greater selfconcept also leads to a reduction in the gender and age gaps in the participating private universities with respect to the public counterparts. It is advisable to increase both faculty training in the didactic use of virtual reality and funding for its use, mainly in public universities. Full article

(This article belongs to the Special Issue Feature Papers in Multimodal Technologies and Interaction—Edition 2023)

► Show Figures

Figure 1

17 pages, 3070 KiB

Open AccessArticle

Understanding Operator Influence in Automated Urban Shuttle Buses and Recommendations for Future Development

by Martina Schuß, Alice Rollwagen and Andreas Riener

Multimodal Technol. Interact. 2022, 6(12), 109; https://doi.org/10.3390/mti6120109 - 13 Dec 2022

Cited by 2 | Viewed by 2023

Abstract

The automation of our vehicles is an all-present topic with great benefits for society, particularly in the area of public transport and pilot projects of automated shuttle buses are already underway. However, they do not show the full potential of using them as [...] Read more.

The automation of our vehicles is an all-present topic with great benefits for society, particularly in the area of public transport and pilot projects of automated shuttle buses are already underway. However, they do not show the full potential of using them as a supplement to public transport, since single-occupancy registration of the vehicles usually allows only slow speeds and also requires a substitute driver on board. In our study, we aim to (1) examine the status quo of its user acceptance and (2) identify the roles of the operators and their tasks in automated urban shuttle buses. We conducted a mixed-method study including in-depth interviews, questionnaires, and in-the-field observations visiting pilot projects of the two most widespread pilot projects on German streets. Our results uncover the multiple roles and tasks the human operators currently assume. Furthermore, we developed design approaches for a digital companion substituting the operator in a long run and evaluated these concepts. A remote operator or a hologram were preferred solutions and we propose further design requirements for such companions. This work helps to understand the individual roles that operators currently occupy and provides a good basis for concepts of technologies that will perform these tasks in the future. Full article

(This article belongs to the Special Issue Interaction Design and the Automated City – Emerging Urban Interfaces, Prototyping Approaches and Design Methods)

► Show Figures

Figure 1

16 pages, 3699 KiB

Open AccessArticle

Usability Tests for Texture Comparison in an Electroadhesion-Based Haptic Device

by Afonso Castiço and Paulo Cardoso

Multimodal Technol. Interact. 2022, 6(12), 108; https://doi.org/10.3390/mti6120108 - 08 Dec 2022

Cited by 2 | Viewed by 1369

Abstract

Haptic displays have been gaining more relevance over the recent years, in part because of the multiple advantages they present compared with standard displays, especially for improved user experience and their many different fields of application. Among the various haptic technologies, electroadhesion is [...] Read more.

Haptic displays have been gaining more relevance over the recent years, in part because of the multiple advantages they present compared with standard displays, especially for improved user experience and their many different fields of application. Among the various haptic technologies, electroadhesion is seen as capable of better interaction with a user, through a display. TanvasTouch is an economically competitive haptic device using electroadhesion, providing an API and respective haptic engine, which makes the development of applications much easier and more systematic than in the past, back when the creation of these haptic solutions required a greater amount of work and resulted in ad-hoc solutions. Despite these advantages, it is important to access its ability to describe textures in a way understandable by the user’s touch. The current paper presents a set of experiments using TanvasTouch electroadhesion-based haptic technology to access how a texture created on a TanvasTouch device can be perceived as a representation of a real-world object. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives)

► Show Figures

Figure 1

18 pages, 5519 KiB

Open AccessArticle

User Authentication Recognition Process Using Long Short-Term Memory Model

by Bengie L. Ortiz, Vibhuti Gupta, Jo Woon Chong, Kwanghee Jung and Tim Dallas

Multimodal Technol. Interact. 2022, 6(12), 107; https://doi.org/10.3390/mti6120107 - 30 Nov 2022

Cited by 1 | Viewed by 1877

Abstract

User authentication (UA) is the process by which biometric techniques are used by a person to gain access to a physical or virtual site. UA has been implemented in various applications such as financial transactions, data privacy, and access control. Various techniques, such [...] Read more.

User authentication (UA) is the process by which biometric techniques are used by a person to gain access to a physical or virtual site. UA has been implemented in various applications such as financial transactions, data privacy, and access control. Various techniques, such as facial and fingerprint recognition, have been proposed for healthcare monitoring to address biometric recognition problems. Photoplethysmography (PPG) technology is an optical sensing technique which collects volumetric blood change data from the subject’s skin near the fingertips, earlobes, or forehead. PPG signals can be readily acquired from devices such as smartphones, smartwatches, or web cameras. Classical machine learning techniques, such as decision trees, support vector machine (SVM), and k-nearest neighbor (kNN), have been proposed for PPG identification. We developed a UA classification method for smart devices using long short-term memory (LSTM). Specifically, our UA classifier algorithm uses raw signals so as not to lose the specific characteristics of the PPG signal coming from each user’s specific behavior. In the UA context, false positive and false negative rates are crucial. We recruited thirty healthy subjects and used a smartphone to take PPG data. Experimental results show that our Bi-LSTM-based UA algorithm based on the feature-based machine learning and raw data-based deep learning approaches provides 95.0% and 96.7% accuracy, respectively. Full article

► Show Figures

Figure 1

18 pages, 3078 KiB

Open AccessArticle

Inhibitors and Enablers to Explainable AI Success: A Systematic Examination of Explanation Complexity and Individual Characteristics

by Carolin Wienrich, Astrid Carolus, David Roth-Isigkeit and Andreas Hotho

Multimodal Technol. Interact. 2022, 6(12), 106; https://doi.org/10.3390/mti6120106 - 28 Nov 2022

Cited by 1 | Viewed by 1707

Abstract

With the increasing adaptability and complexity of advisory artificial intelligence (AI)-based agents, the topics of explainable AI and human-centered AI are moving close together. Variations in the explanation itself have been widely studied, with some contradictory results. These could be due to users’ [...] Read more.

With the increasing adaptability and complexity of advisory artificial intelligence (AI)-based agents, the topics of explainable AI and human-centered AI are moving close together. Variations in the explanation itself have been widely studied, with some contradictory results. These could be due to users’ individual differences, which have rarely been systematically studied regarding their inhibiting or enabling effect on the fulfillment of explanation objectives (such as trust, understanding, or workload). This paper aims to shed light on the significance of human dimensions (gender, age, trust disposition, need for cognition, affinity for technology, self-efficacy, attitudes, and mind attribution) as well as their interplay with different explanation modes (no, simple, or complex explanation). Participants played the game Deal or No Deal while interacting with an AI-based agent. The agent gave advice to the participants on whether they should accept or reject the deals offered to them. As expected, giving an explanation had a positive influence on the explanation objectives. However, the users’ individual characteristics particularly reinforced the fulfillment of the objectives. The strongest predictor of objective fulfillment was the degree of attribution of human characteristics. The more human characteristics were attributed, the more trust was placed in the agent, advice was more likely to be accepted and understood, and important needs were satisfied during the interaction. Thus, the current work contributes to a better understanding of the design of explanations of an AI-based agent system that takes into account individual characteristics and meets the demand for both explainable and human-centered agent systems. Full article

(This article belongs to the Special Issue Multimodal Conversational Interaction and Interfaces, Volume II)

► Show Figures

Figure 1

18 pages, 953 KiB

Open AccessArticle

Narrative Visualization with Augmented Reality

by Ana Beatriz Marques, Vasco Branco and Rui Costa

Multimodal Technol. Interact. 2022, 6(12), 105; https://doi.org/10.3390/mti6120105 - 26 Nov 2022

Cited by 1 | Viewed by 2564

Abstract

The following study addresses, from a design perspective, narrative visualization using augmented reality (AR) in real physical spaces, and specifically in spaces with no semantic relation with the represented data. We intend to identify the aspects augmented reality adds, as narrative possibilities, to [...] Read more.

The following study addresses, from a design perspective, narrative visualization using augmented reality (AR) in real physical spaces, and specifically in spaces with no semantic relation with the represented data. We intend to identify the aspects augmented reality adds, as narrative possibilities, to data visualization. Particularly, we seek to identify the aspects augmented reality introduces regarding the three dimensions of narrative visualization—view, focus and sequence. For this purpose, we adopted a comparative analysis of a set of fifty case studies, specifically, narrative visualizations using augmented reality from a journalistic scope, where narrative is a key feature. Despite the strong explanatory character that characterizes the set of analyzed cases, which sometimes limits the user’s agency, there is a strong interactive factor. It was found that augmented reality can expand the narrative possibilities in the three dimensions mentioned—view, focus and sequence—but especially regarding visual strategies where simulation plays an essential role. As a visual strategy, simulation can provide the context for communication or be the object of communication itself, as a replica. Full article

(This article belongs to the Special Issue Virtual Reality and Augmented Reality)

► Show Figures

Figure 1

19 pages, 8339 KiB

Open AccessArticle

Design and Implementation of a Personalizable Alternative Mouse and Keyboard Interface for Individuals with Limited Upper Limb Mobility

by Daniel Andreas, Hannah Six, Adna Bliek and Philipp Beckerle

Multimodal Technol. Interact. 2022, 6(12), 104; https://doi.org/10.3390/mti6120104 - 24 Nov 2022

Viewed by 1817

Abstract

People with neuromuscular diseases often experience limited upper limb mobility, which makes the handling of standard computer mice and keyboards difficult. Due to the importance of computers in private and professional life, this work aims at implementing an alternative mouse and keyboard interface [...] Read more.

People with neuromuscular diseases often experience limited upper limb mobility, which makes the handling of standard computer mice and keyboards difficult. Due to the importance of computers in private and professional life, this work aims at implementing an alternative mouse and keyboard interface that will allow for their efficient use by people with a neuromuscular disease. Due to the strongly differing symptoms of these diseases, personalization on the hardware and software levels is the focus of our work. The presented mouse alternative is based on a spectacle frame with an integrated motion sensor for head tracking, which enables the control of the mouse cursor position; the keyboard alternative consists of ten keys, which are used to generate word suggestions for the user input. The interface was tested in a user study involving three participants without disabilities, which showed the general functionality of the system and potential room for improvement. With an average throughput of

1.56

bits per second achieved by the alternative mouse and typing speeds of

8.44

words per minute obtained using the alternative keyboard, the proposed interface could be a promising input device for people with limited upper limb mobility. Full article

(This article belongs to the Special Issue Feature Papers in Multimodal Technologies and Interaction—Edition 2023)

► Show Figures

Figure 1

18 pages, 2483 KiB

Open AccessArticle

Identifying and Addressing Needs of Heterogeneous User Groups—A Case Study from the Banking Sector

by Kathrin Pollmann, Doris Janssen, Nora Fronemann, Milena Velić, Philippe Bouillé, Béatrice Foucault and Nathalie Bégoc Bécam

Multimodal Technol. Interact. 2022, 6(12), 103; https://doi.org/10.3390/mti6120103 - 23 Nov 2022

Cited by 1 | Viewed by 1415

Abstract

Adopting a need-based approach can help companies to create products and services that are preferred by their customers and improve their well-being, thus providing a competitive advantage. To put need-based designs into practice, it would be interesting to know how innovative product and [...] Read more.

Adopting a need-based approach can help companies to create products and services that are preferred by their customers and improve their well-being, thus providing a competitive advantage. To put need-based designs into practice, it would be interesting to know how innovative product and service ideas can address the needs of a specific target group in a specific business domain. This paper presents an approach for (a) identifying such target group-specific need sets based on an online survey and (b) integrating them into the company’s innovation processes as part of a Need-based Creativity Workshop. To illustrate and validate this approach, we present a case study that investigates varying need subsets for two different user groups of future banking products and services in France: adults with and without families. Our study shows that a different set of needs is important for each group and reflects upon the benefits and challenges of a target group-specific, need-based design approach to leverage a company’s potential for innovation. Full article

(This article belongs to the Special Issue Design for Wellbeing at Scale)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 6, Issue 12 (December 2022) – 11 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI