Skip to main content

2022 | Buch

Computers Helping People with Special Needs

18th International Conference, ICCHP-AAATE 2022, Lecco, Italy, July 11–15, 2022, Proceedings, Part I

herausgegeben von: Klaus Miesenberger, Georgios Kouroupetroglou, Katerina Mavrou, Roberto Manduchi, Dr. Mario Covarrubias Rodriguez, Petr Penáz

Verlag: Springer International Publishing

Buchreihe : Lecture Notes in Computer Science

insite
SUCHEN

Über dieses Buch

The two-volume set LNCS 13341 and 13342 constitutes the refereed proceedings of the Joint International Conference on Digital Inclusion, Assistive Technology, and Accessibility, ICCHP-AAATE 2022. The conference was held in Lecco, Italy, in July 2022.

The 112 papers presented were carefully reviewed and selected from 285 submissions. Included also are 18 introductions. The papers are organized in the following topical sections:

Part I: Art Karshmer Lectures in Access to Mathematics, Science and Engineering; Digital Solutions for Inclusive Mobility: solutions and accessible maps for indoor and outdoor mobility; implementation and innovation in the area of independent mobility through digital technologies; haptic and digital access to art and artefacts; accessibility of co-located meetings; interactions for text input and alternative pointing; cognitive disabilities and accessibility; augmentative and alternative communication (AAC): emerging trends, opportunities and innovations; language accessibility for the deaf and hard-of-hearing.

Part II: Digital accessibility: readability and understandability; serious and fun games; internet of things: services and applications for people with disabilities and elderly persons; technologies for inclusion and participation at work and everyday activities; robotic and virtual reality technologies for children with disabilities and older adults; development, evaluation and assessment of assistive technologies; ICT to support inclusive education – universal learning design (ULD); design for assistive technologies and rehabilitation; assistive technologies and inclusion for older people.

Inhaltsverzeichnis

Frontmatter

Art Karshmer Lectures in Access to Mathematics, Science and Engineering

Frontmatter
Art Karshmer Lectures in Access to Mathematics, Science and Engineering
Introduction to the Special Thematic Session

Pr Art Karshmer started this series of sessions about access to STEM by people with visual impairment 20 years ago, aiming at presenting the state of the art of researches dedicated to non visual access to scientific content and calculation. This year we will read ten papers presenting research works using machine learning, but also computer algebra systems and natural language processing, to help blind and partially sighted users to access to scientific content, to understand them and finally to learn mathematics and to do calculations. Also, four of these papers concern access to graphical content.

Dominique Archambault, Katsuhito Yamaguchi, Georgios Kouroupetroglou, Klaus Miesenberger

Open Access

Conversion of Multi-lingual STEM Documents in E-Born PDF into Various Accessible E-Formats

A new method of mathematical OCR to improve remarkably recognition accuracy for e-born PDF by making use of SVG information generated from the PDF is shown. Even if a local language is used to represent texts in the PDF, without a special OCR engine for that language, it can be converted into various accessible e-formats. The software GUI is improved so that end users can customize it easily for their language. Its French and Vietnamese versions are actually released by using this new feature. Some evaluations done in Vietnam are also reported.

Masakazu Suzuki, Katsuhito Yamaguchi

Open Access

An Efficient Method to Produce Accessible Contents for Online STEM Education

Many online educational materials for STEM are now produced with Microsoft PowerPoint (PPT); however, many of them are not necessarily accessible for print-disabled students. By making use of Infty software, our new add-on allows users to add easily alt text/aloud reading with a TTS voice to any technical part such as math expressions included in PPT slides. An accessible MP4 video for STEM education also can be produced efficiently. The video contents produced in this manner are preliminarily evaluated by comparison with another accessible content: “ChattyBook" (audio-embedded HTML5).

Toshihiko Komada, Katsuhito Yamaguchi, Masakazu Suzuki
Making Equations Accessible in Scientific Documents

Unlike a standard text document, a STEM document not only consists of text information but different components such as tables, figures, captions, mathematical equations etc. This paper presents a novel technique to detect mathematical equations in PDF documents and convert those equations into a more accessible format such as Latex . We use visual features of the document to detect the mathematical equations using object detection and subsequently apply heuristics to the generated bounding boxes to precisely cover the complete equation. These detections are passed to a tool called Maxtract which will rewrite the equations in Latex .

Shivansh Juyal, Sanjeev Sharma, Neha Jadhav, Volker Sorge, M. Balakrishnan
Towards Semantically Enhanced Audio Rendering of Equations

At present STEM education beyond middle school is largely inaccessible to visually impaired students in countries like India. Access to equations, tables, charts and figures are the key bottleneck. With the increased penetration of screen reading software, effective audio rendering of equations can significantly help in making many of the e-texts accessible. Unfortunately, linear syntactic rendering of equations can create considerable cognitive load even for relatively simple equations. In this paper we propose an architecture to extract contextual semantic of equations based on the local definitions. This will help in adapting audio rendering of equations based on their contextual semantics.

Akashdeep Bansal, Volker Sorge, M. Balakrishnan, Aayush Aggarwal
Accessible Chemical Structural Formulas Through Interactive Document Labeling

Despite a number of advances in the accessibility of STEM education, there is a lack of advanced tool support for authors and educators seeking to make corresponding documents accessible. We propose an interactive labeling method that combines an AI with user input to create accessible chemical structural formulas and incrementally improve the model. The model is a deep learning method based on a convolutional neural network and a transformer-based encoder-decoder. We implement this in a tool that enables graphical labeling of structural formulas and supports the user by performing a similarity search to suggest matches. Our approach aims to improve both the efficiency and effectiveness of labeling chemical structural formulas for accessibility purposes.

Merlin Knaeble, Zihan Chen, Thorsten Schwarz, Gabriel Sailer, Kailun Yang, Rainer Stiefelhagen, Alexander Maedche
Designing an Inclusive and Accessible Mathematical Learning Environment Based on a Theorem Prover

A novel approach to design an inclusive and accessible mathematical learning environment is presented: The technology of theorem proving shall be employed to support a student in solving mathematical problems by giving hints to him/her based on formal proofs of each step in a calculation. The system shall be made accessible by making use of the built-in accessibility coming with VSCode, a standard editor used as front-end for the theorem prover Isabelle.

Bernhard Stöger, Klaus Miesenberger, Walther Neuper, Makarius Wenzel, Thomas Neumayr
Developing a Corpus of Hierarchically Classified STEM Images for Accessibility Purposes

Even though considerable efforts have been made to provide effective image descriptions for digital accessibility, a large portion of STEM images, especially complex STEM images, nowadays remains inaccessible to people with visual disability. The quality of alt text is much more critical for university STEM textbooks as image descriptions must be accurate and detailed but not tire out the reader. This work aims to develop a large corpus of hierarchically classified STEM images from university textbooks which later will be used for developing appropriate guidelines for meaningful non-automatic high-quality alt-text image descriptions with the purpose of accessibility in mind. We present first our approach for the creation of the corpus with the STEM images. Our corpus at the current stage includes 8.859 STEM images from 82 textbooks in the domains of Mathematics, Biology, Computer Science, Chemistry, Physics and Geology. Then, we describe the methodology we followed for the classification of the images in the corpus and in particular, the way for the creation of the five categories and twenty-four subcategories, as well as the manner of the assignment of images to categories.

Theodora Antonakopoulou, Paraskevi Riga, Georgios Kouroupetroglou

Open Access

Effective Non-visual Access to Diagrams via an Augmented Natural Language Interface

This paper describes the design and validation of a number of HCI techniques that enable more effective non-visual access to diagrammatically displayed data through an adapted Natural Language Interface (NLI). These techniques have been incorporated into an accessible web-based NLI to diagrams prototype and qualitatively evaluated with blind persons for solving disparate analytical and navigational tasks supported by diagrams. Each technique is described and validation results discussed.

Tomas Murillo-Morales, Klaus Miesenberger
Interface for Automatic Tactile Display of Data Plots

Graphical representation of mathematical functions is often inaccessible to blind people. There are methods to convert graphs into a tactile form, but this requires assistance, apart from manufacturing time and cost. In this paper, we present a system that enables blind people to independently display functions on a tactile display via a computer algebra system. Additional tools have been developed to further analyze a function (switchable axes, coordinate markers and zoom). To test the efficiency of the system, a pilot study was conducted with 5 blind participants. All participants were able to solve the tasks independently and were very positive about the system.

Thorsten Schwarz, Giuseppe Melfi, Stefan Scheiffele, Rainer Stiefelhagen
Mobile e-Learning Platform for Audio-Tactile Graphics Presentation

The use of multimedia in education means that more and more information is presented graphically. However, it limits the education of blind people, especially in science, where graphical content is the most often way of presenting the information. One of the commonly used alternative forms of graphic presentation is tactile graphics with descriptions in braille. Nonetheless, in the case of complex pictures, this form of presentation is insufficient for a blind student, and reading the tactile pictures often requires an assistant’s help. In the paper, we present a developed e-learning platform for audio-tactile graphics presentation for the blinds dedicated to standard Android tablets, whose advantage is the automatic adaptation of interactive audio descriptions to the student’s level of knowledge. The platform also enables the automatic assessment of the student's knowledge and exercise selection assigned to the appropriate knowledge level. The paper also presents the research results regarding the effectiveness of self-learning using our platform on two age groups of blind students while solving math exercises. The obtained research results show that the developed platform and the proposed method of verifying students’ knowledge and selecting exercises positively influence the improvement of learning effectiveness. The platform can be used during the traditional lesson at school and also for self-learning which makes it especially desirable during the Coronavirus pandemic.

Michał Maćkowski, Piotr Brzoza, Mateusz Kawulok, Tomasz Knura

Digital Solutions for Inclusive Mobility: Solutions and Accessible Maps for Indoor and Outdoor Mobility

Frontmatter
Digital Solutions for Inclusive Mobility: Solutions and Accessible Maps for Indoor and Outdoor Mobility
Introduction to the Special Thematic Session

The current paper is an introduction to the Special Thematic Session Digital Solutions for Inclusive Mobility: Solutions and Accessible Maps for Indoor and Outdoor Mobility. Mobility is a valuable asset that should be equally available to all people. This chapter addresses the following problems which need to be solved so that people with disabilities can also enjoy greater mobility: getting the relevant information about buildings, orientating oneself in buildings, accessing tactile indoor maps, providing navigation in indoor and outdoor environments, tailoring the route to the users, creating accessible navigation apps, and making them adaptable to the users’ needs.

Claudia Loitsch, Karin Müller, Gerhard Weber, Helen Petrie, Rainer Stiefelhagen
Split it Up: Allocentric Descriptions of Indoor Maps for People with Visual Impairments

Planning a trip to unfamiliar public buildings is challenging for people with visual impairments as much visual information such as floor plans is not accessible. Textual descriptions of an indoor map would therefore be very useful to prepare a trip. In particular, an allocentric description independent from the current location of the user which could be used at home would support the preparation phase. However, descriptions of buildings are rarely available. So the main question is how to tailor the descriptions to the needs of people with visual impairments in order to not overwhelm them with too much information at a time. We propose a system for the generation of allocentric textual descriptions for public buildings. In a user study, we tested the usefulness of our system and found that a modular design is regarded helpful to clearly structure descriptions by splitting the description into meaningful modules and avoiding information overload.

Julia Anken, Danilo Rosenthal, Karin Müller, Gerhard Jaworek, Rainer Stiefelhagen
Expert Study: Design and Use of Textures for Tactile Indoor Maps with Varying Elevation Levels

Tactile maps could increase the mobility of people with blindness and visual impairments. Most research on the design and development of tactile maps focuses on outdoor environments, whereby indoor environments greatly differ in terms of represented objects, structure, information, and purpose of use. Studies on the design and use of tactile indoor maps are missing, so it is still unclear how textures can be used effectively in tactile maps and which are suitable for which areas and contexts. As a first step, we therefore conducted an expert user study with four blind participants to determine the challenges and benefits of textures in tactile indoor maps. Afterward, we initiated a follow-up study with two experts from the initial study to evaluate the influence of different elevation levels and production methods (swell paper and embossed maps) on the recognition of different textures, symbols, Braille labels, and structure of the building. We observed an influence of the elevation level of the texture on the recognition of map elements. Furthermore, the perceived structure of the texture influences the recognizability of the map elements. As a result, we identified suitable types of textures for rooms and floors in tactile indoor maps.

Christin Engel, Gerhard Weber
ATIM: Automated Generation of Interactive, Audio-Tactile Indoor Maps by Means of a Digital Pen

Mobile independence requires knowledge about the environment and environmental features. In particular, the location and structure of objects in the environment must be known in order to navigate independently. Tactile maps often require Braille skills as well as tactile graphic reading skills, and also contain less details or need to be divided into many sheets. Audio-tactile maps can overcome these disadvantages and can be used effectively with tactile indoor maps. In this paper, we first classify user tasks of tactile indoor maps. We further present an initial prototype that enables the automated generation of audio-tactile indoor maps on the basis of OSM data that could be used with a digital pen and a smartphone application to get detailed information about the building. It further provides search mode that supports finding a target in a building with sonification. We implemented multiple gestures to interact with the graphic and investigate an initial pilot study with one person with blindness to get first qualitative feedback in terms of our concept. The implemented workflow enables the accessible creation of low-cost, mobile and interactive audio-tactile indoor maps by means with a digital pen.

Christin Engel, Gerhard Weber
An Audio-Tactile System for Visually Impaired People to Explore Indoor Maps

Nowadays, a great amount of information is communicated in visual form. This excludes visually impaired people from easily accessing that information. A consequence is that they tend to limit their mobility due to lack of information. The goal of this work is to make indoor maps more accessible to help blind people preparing a trip to unknown buildings. In our approach, we further developed an interactive audio-tactile system to facilitate access to indoor maps. A study with blind participants showed that the prototype was well accepted and easy to use. The participants were able to achieve accurate mental models of the provided maps. A further comparison with sighted participants using visual maps showed no significant differences in the ability to describe the maps.

Giuseppe Melfi, Jean Baumgarten, Karin Müller, Rainer Stiefelhagen
Supporting Independent Travelling for People with Visual Impairments in Buildings by Harmonizing Maps on Embossed Paper and Pin-Matrix Devices for Accessible Info-Points

Orientation in unknown buildings is a grand challenge for people with visual impairments in planned as well as in spontaneous scenarios. Therefore, it is necessary to develop solutions that support the whole travelling chain - on the one side the planning process for trips at home and on the other side the orientation process at a building. While different approaches exist for both contexts, in this paper we present a concept using two approaches together for planning and carrying out trips to unknown buildings. We focus on the one hand on tactile printed or embossed maps and on the other hand on digital tactile pin-matrix displays. The first type is suitable for pre-journey planning activities, while the second technology is designed for on-site use. Both approaches are able to provide spatial and configurational knowledge for the tactile sense. Based on this, we describe in this paper how both types could be harmonized to facilitate access to indoor maps in the given scenarios. We present prototypes for tactile indoor maps on embossed paper and demonstrate how a interaction on pin-matrix device can be designed and implemented to be used together and allow similar interaction with both.We propose providing dynamic maps as accessible info-points in buildings. Special focus is on the challenge of splitting tactile indoor maps into multiple views or sheets and how this can be achieved in a similar way with both methods.

Jan Schmalfuß-Schwarz, Christin Engel, Gerhard Weber
The Accessible Tactile Indoor Maps (ATIM) Symbol Set: A Common Symbol Set for Different Printing Methods

In this paper, we describe a method on how to create a common tactile symbol set for printing methods which differ in resolution. We used a well-tested symbol set for swell paper as a basis and defined criteria on how to transfer the symbols to a different printing method. The method was developed in a user-centered design approach and assessed by a blind expert on tactile graphics resulting in the Accessible Tactile Indoor Maps (ATIM) symbol set. Moreover, we extracted the most important accessibility features for indoor environments and mapped them to distinct symbols. The presented method can also be useful for transfer to other printing techniques with a different resolution.

Giuseppe Melfi, Karin Müller, Gerhard Jaworek, Rainer Stiefelhagen
Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor

Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated. After the ego-motion estimation, the dynamic object can be identified and tracked. Then, poses and speed of tracked dynamic objects can be estimated, which are passed to the users through acoustic feedback.

Wenyan Ou, Jiaming Zhang, Kunyu Peng, Kailun Yang, Gerhard Jaworek, Karin Müller, Rainer Stiefelhagen
Accessible Adaptable Indoor Routing for People with Disabilities

For people with disabilities, indoor routing approaches have to take the specific requirements of the target user group into account. Depending on the needs of the individual, certain objects and indoor features can present insurmountable barriers and hence, should be avoided when generating indoor routes. Research in the field of indoor routing for people with disabilities has been going on for several years, but most approaches focus on one specific disability and do not evaluate designed systems with the target user group. Therefore, we propose an accessible, adaptable indoor routing algorithm for people with disabilities. The designed system is evaluated in a user study with people with blindness and mobility impairments using a Wizard of Oz approach. Results indicate a good acceptance of the designed routing system.

Fabian Lüders, Julian Striegl, Jan Schmalfuß-Schwarz, Claudia Loitsch, Gerhard Weber
Monocular Localization Using Invariant Image Feature Matching to Assist Navigation

Indoor positioning is critical for applications like navigation, tracking, monitoring, and accessibility. For the visually impaired this has a huge implication on independent mobility for accessing all types of services as well as social inclusion. The unavailability of indoor positioning solutions with adequate accuracy is a major constraint. The key reason for the lack of growth in indoor positioning systems is to do with the reliability of indoor positioning techniques and additional infrastructure costs along with maintenance overheads. We propose a novel single camera-based visual positioning solution for indoor spaces. Our method uses smart visual feature selection and matching in real-time using a monocular camera. We record and transform the video route information into spars and invariant point-based SURF features. To limit the real-time feature search and match data, the routes inside the buildings are broken into a connected graph. To find the position, confidence of a path increases if it founds a good feature match and decreases otherwise. Each query frame uses a K-nearest neighbor match with the existing databases to increase the confidence of matched path in subsequent frames. Results have shown a reliable positioning accuracy of $$\sim $$ ∼ 2 meters in variable lighting conditions. We also investigated the error recovery of positioning systems where it easily re-positions the user within the neighboring edges. To promote crowdsourcing, proposed system can add more visual features to the database while performing the matching task.

Vikas Upadhyay, M. Balakrishnan
Can Route Previews Amplify Building Orientation for People with Visual Impairment?

Before visiting a new location, we often look at maps to build an a priori mental representation of the environment using landmarks and points of interest. No such options are easily available for persons with visual impairment (PVIs). As a result, PVIs are often dependent on others for assistance with any activities that require visiting unfamiliar spaces. This seriously constrains PVIs’ social inclusion and ability and right to independently explore their environment. From our earlier studies with experienced PVIs, we explored existing way-finding strategies and the role of environmental cues in aiding orientation and mobility. Here, we build on these findings to propose a route preview assistance application for PVIs to improve building orientation by helping to create a mental representation of spaces they are planning to visit. We translated non-visual cues into potential landmarks and encoded them into a mapping tool - IncluMap. The mapped building data is used to generate an aural route preview. In the follow-up study with five PVI participants, we evaluated how route preview assistance can improve orientation and route knowledge. We show that even for longer routes (>50 m) with a relatively good number of cognitive landmarks (>5), participants were able to remember and build a mental map. We found cognitive landmarks were useful for PVIs to support re-localization and getting reassurance in complex indoor spaces. Based on these findings we believe that the proposed route preview assistance can be a useful tool to provide orientation and mobility training to PVIs.

Vikas Upadhyay, Tigmanshu Bhatnagar, Catherine Holloway, P. V. M. Rao, M. Balakrishnan

Implementation and Innovation in the Area of Independent Mobility Through Digital Technologies

Frontmatter
Implementation and Innovation in the Area of Independent Mobility Through Digital Technologies
Introduction to the Special Thematic Session

This STS explores contributions to the independent mobility of persons with disabilities. Mobility is often cited as a requirement of people with a disability seeking to increase opportunities for full access to education, employment and daily living. Recent innovations in assistive and accessible technologies suggest new ways in which navigation, orientation, and wayfinding can be accessible for people with a wide variety of needs, including those with cognitive, sensory, and physical impairments and the elderly.The papers share insight and examples of the impact of emerging technologies, including location-based technologies, artificial intelligence, machine learning, and augmented mixed and virtual reality solutions. These offer opportunities for increasingly personalised, adaptive, and natural interfaces for access methods, including speech, audio, gestural and haptic approaches.Such innovation is often based upon co-design requires appropriate service and training to be implemented across age, needs, and context. The STS discusses recent achievements in independent mobility through digital technologies, focusing on the technologies, forms of access, and associated services to support implementation, replication, and localisation.

David Banes

Open Access

The Impact of Subjective Technology Adaptivity on the Willingness of Persons with Disabilities to Use Emerging Assistive Technologies: A European Perspective

Emerging digital technologies like augmented reality (AR) hold promising prospects for people with disabilities. It remains, however, an open question how persons with disabilities respond to technological demands. The paper examines the potential impact of users’ self-assessment of their own competence in using these technologies on users’ responses by examining their Subjective Technology Adaptivity (STA) [1] and use intention to study the relationship between their self-assessed adaptivity and volitional technology use. To this end, data from 545 Europeans with different types of disabilities were collected based on an online survey. The research focused on six emerging assistive technologies related to mobility: accessible navigation systems, artificial intelligence alerts, wearables, robots, augmented reality and location-based alerts. The results show that the adaptivity to technology of people with disabilities predicts the use intention for emerging assistive technologies. There was, however, great variability depending on the type of disability. For example, a high STA of people with physical, visual, hearing or intellectual impairments predicted their willingness to use intention of AI-based alters but not for people with mental health issues or multiple impairments. Our findings shed new light on the role of perceived technology adaptivity of persons with disabilities for future technology use intention.

Alexandra König, Laura Alčiauskaitė, Tally Hatzakis
Towards Personalized Accessible Routing for People with Mobility Impairments

This paper presents a vision for personalized accessible routing for people with mobility impairments and discusses which accessibility information is needed to realize this vision, such as curb cuts, inclination, path width, and additional information about various barriers. In a case study for the city of zurich, the paper first analyses how much of this accessibility information is available and in what quality in OpenStreetMap (OSM), and then investigates different approaches for collecting the missing accessibility data. This accessibility data can be used to implement novel routing apps for pedestrians with mobility impairments and can support city planners in considering future measures to improve urban accessibility. The main questions of this research are: What accessibility information is needed to allow for a personalized accessible routing? How can the missing accessibility data be collected efficiently? We suggest a methodology consisting of a combination of automated, semi-automated, crowd-based, and onsite data collection.

Alireza Darvishy, Hans-Peter Hutter, Roland Mosimann
Traveling to Unknown Buildings: Accessibility Features for Indoor Maps

Traveling independently to unknown buildings is difficult for people with disabilities, as there is a lack of information about accessibility of indoor environments. In particular, there are hardly any freely available indoor maps of public buildings. In this paper, we address the problem that there is no comprehensive list of information relevant to people with disabilities in indoor environments, which in turn can be used for indoor orientation and navigation systems.We therefore collected in an extensive literature review around 820 indoor accessibility features relevant for people with disabilities. These were categorized and sorted in a database and mapped to OSM and A11yJSON. The database is publicly available and can serve as a basis for tag proposals to OSM, and as Linked Data (RDF).

Angela Constantinescu, Karin Müller, Claudia Loitsch, Sebastian Zappe, Rainer Stiefelhagen
Acquiring Surrounding Visual Information Without Actively Taking Photos for People with Visual Impairment

Recent advancements in recognition technology allow people with visual impairment to obtain visual information from their surroundings using smartphone apps and assistive devices. This paper points out a problem with this approach that has not attracted much attention. That is, the user is required to actively take a photo, which is not always easy for people with visual impairment. To address this problem, in contrast to the current standard approach, which we call active information acquisition, we propose passive information acquisition (PIA), which does not require the user to actively take a photo. However, PIA creates a new problem: the app tends to transfer too much information to the user. Therefore, this paper explores better ways for people with visual impairment toward obtaining only the desired visual information in PIA. Specifically, we experimented with nine people with visual impairment to evaluate seven information transmission methods, including information summarization and interactive communication methods.

Masakazu Iwamura, Takaaki Kawai, Keigo Takashima, Kazunori Minatani, Koichi Kise
Listening First: Egocentric Textual Descriptions of Indoor Spaces for People with Blindness

Orientation in unknown environments is challenging for people with. Especially in indoor environments, there are very few systems that support navigation but almost none for orientation. Thus, we propose a grammar for generating German textual descriptions of indoor environments for users with blindness. We utilize an egocentric approach taking into account the user’s location and orientation in the building. To investigate what word order is preferred, we compare descriptions generated by three different grammars. We also examine what strategies people with blindness pursue to orient themselves in buildings. We test our concept in an online user study with people with blindness. Our study shows that egocentric information should be brief, always following the same structure and allow for customization.

Angela Constantinescu, Eva-Maria Neumann, Karin Müller, Gerhard Jaworek, Rainer Stiefelhagen

Haptic and Digital Access to Art and Artefacts

Frontmatter
Non-visual Access to an Interactive 3D Map

Maps are indispensable for helping people learn about unfamiliar environments and plan trips. While tactile (2D) and 3D maps offer non-visual map access to people who are blind or visually impaired (BVI), this access is greatly enhanced by adding interactivity to the maps: when the user points at a feature of interest on the map, the name and other information about the feature is read aloud in audio. We explore how the use of an interactive 3D map of a playground, containing over seventy play structures and other features, affects spatial learning and cognition. Specifically, we perform experiments in which four blind participants answer questions about the map to evaluate their grasp of three types of spatial knowledge: landmark, route and survey. The results of these experiments demonstrate that participants are able to acquire this knowledge, most of which would be inaccessible without the interactivity of the map.

James M. Coughlan, Brandon Biggs, Huiying Shen

Open Access

Development of Tabletop Models of Internal Organs for Anatomy Learning of the Visually Impaired

In this study, we developed two types of tabletop models of internal organs as tactile teaching materials that can be used by the visually impaired when learning anatomy. The first one is a model in which parts of the internal organs that do not have connectional relationships are placed in a concave area of a pedestal. The second one is a model in which parts of the internal organs have a connectional relationship. Parts of the organs were connected using connecting pins, holes, and rubber tubes. A connected respiratory system, a connected digestive system, and a connected urinary system are also placed in the concave areas of the pedestal. Evaluation experiments conducted on the two types of tactile teaching materials revealed that the second model is a better teaching material in terms of placement stability and an understanding of the connectional relationship of internal organs.

Moeka Shinoda, Akihiro Koike, Sayaka Teraguchi, Yoshinori Teshima
Semi-automatic Contour “Gist” Creation for Museum Painting Tactile Exploration

Contour “gist” creation (i.e., creating a simplified contour which represents an object) is always required in helping Visually Impaired People (VIP) to do an effective tactile exploration on painting images in a museum. However, this process is very labor intensive, and the existing contour/edge detection algorithms in the literature are not capable of creating the contour “gist” models automatically. In this paper, a method for semi-automatic contour “gist” creation for museum painting tactile exploration is proposed. It uses 2 databases (original image database and contour “gist” model database), an object detection algorithm (deep learning), and object-contour “gist” model matching algorithm. The output of this method would be transferred to Force-Feedback Tablet (F2T) (an original tactile device) to allow the VIP to explore the museum paintings more effectively.

Son Duy Dao, Ngoc-Tan Truong, Edwige Pissaloux, Katerine Romeo, Lilia Djoussouf
Inclusive Multimodal Discovery of Cultural Heritage: Listen and Touch

Museums around the world present their collections mainly to visual perception without the possibility of hearing or touching paintings, sculptures and artworks (Size, fragility). It has now become a priority to promote inclusive approaches to issues related to accessibility. Audio and tactile perceptions are often complementary. The figurative elements of the paintings are difficult to describe with words but can be presented tactilely. Other elements cannot be perceived via touch, for example colors, light, some links between elements; but these elements and the feelings they arouse can be evoked via audio-description. We present our tests of objective and sensorial audio description. Then we propose tactile exploration with the use of Force Feedback Tablet (F2T) to discover the general shape of some elements in a painting and complete the perception with an inclusive audio-description.

Katerine Romeo, Hannah Thompson, Marion Chottin

Accessibility of Co-located Meetings

Frontmatter
Accessibility of Co-Located Meetings
Introduction to the Special Thematic Session

Non-verbal communication is an important carrier of information. Even though the spoken word can be heard by blind and visually impaired persons, up to 60% of the overall information still remains inaccessibly to them due to its visual character [11]. However, there is a wide spectrum of non-verbal communication elements, and not all of them are of the same importance. In particular for group meetings, facial expressions and pointing gestures are relevant, which need to be captured, interpreted and output to the blind and visually impaired person.This session first gives a systematic approach to gather the accessibility requirements for blind and visually impaired persons, from which two typical requirements are selected and discussed in more detail. Here, solutions for capturing and interpreting are provided, and finally the session introduces a concept for accessible user interfaces.

Andreas Kunz, Reinhard Koutny, Klaus Miesenberger
Non-verbal Communication and Joint Attention Between People with and Without Visual Impairments: Deriving Guidelines for Inclusive Conversations in Virtual Realities

With the emergence of mainstream virtual reality (VR) platforms for social interactions, non-verbal communicative cues are increasingly being transmitted into the virtual environment. Since VR is primarily a visual medium, accessible VR solutions are required for people with visual impairments (PVI). However, existing propositions do not take into account social interactions, and therefore PVI are excluded from this type of experience. To address this issue, we conducted semi-structured interviews with eleven participants, seven of whom were PVI and four of whom were partners or close friends without visual impairments, to explore how non-verbal cues and joint attention are used and perceived in everyday social situations and conversations. Our goal was to provide guidelines for inclusive conversations in virtual environments for PVI. Our findings suggest that gaze, head direction, head movements, and facial expressions are important for both groups in conversations but often difficult to identify visually for PVI. From our findings, we provide concrete suggestions for the design of social VR spaces, inclusive to PVI.

Markus Wieland, Lauren Thevin, Albrecht Schmidt, Tonja Machulla
Emotion Recognition - A Tool to Improve Meeting Experience for Visually Impaired

Facial expressions play an important role in human communication since they enrich spoken information and help convey additional sentiments e.g. mood. Among others, they non-verbally express a partner’s agreement or disagreement to spoken information. Further, together with the audio signal, humans can even detect nuances of changes in a person’s mood. However, facial expressions remain inaccessible to the blind and visually impaired, and also the voice signal alone might not carry enough mood information.Emotion recognition research mainly focused on detecting one of seven emotion classes. Such emotions are too detailed, and having an overall impression of primary emotional states such as positive, negative, or neutral is more beneficial for the visually impaired person in a lively discussion within a team. Thus, this paper introduces an emotion recognition system that allows a real-time detection of the emotions “agree”, “neutral”, and “disagree”, which are seen as the most important ones during a lively discussion. The proposed system relies on a combination of neural networks that allow extracting emotional states while leveraging the temporal information from videos.

Mathieu Lutfallah, Benno Käch, Christian Hirt, Andreas Kunz
Pointing, Pairing and Grouping Gesture Recognition in Virtual Reality

During a team discussion, participants frequently perform pointing, pairing, or grouping gestures on artifacts on a whiteboard. While the content of the whiteboard is accessible to the blind and visually impaired people, the referring deictic gestures are not. This paper thus introduces an improved algorithm to detect such gestures and to classify them. Since deictic gestures such as pointing, pairing and grouping are performed by sighted users only, we used a VR environment for the development of the gesture recognition algorithm and for the subsequent user studies.

Valentina Gorobets, Cecily Merkle, Andreas Kunz

Open Access

Accessible User Interface Concept for Business Meeting Tool Support Including Spatial and Non-verbal Information for Blind and Visually Impaired People

Business meetings play an essential role in many people’s work life. Although, business meetings have changed over time, especially tools, which are used to support the process, slowly moving from traditional means like flipcharts to more modern, digital alternatives, some aspects stayed the same: Visual information is used to gather thoughts, support arguments and lead the discussion. These kinds of information used to be completely inaccessible to blind and visually impaired people (BVIP) and still are, for the most part. Even though, the movement towards digitalization facilitates accessibility, no fully accessible tool support for business meetings is available. Additionally, non-verbal communication and spatial information is heavily used as well. People use mimic and gestures, and they refer to objects or other people by pointing at them. BVIP miss out on this type of information as well. Ultimately, BVIP have a significant disadvantage during business meetings and very often during their entire professional life. Research efforts have tried to mitigate single aspects of this situation, but no comprehensive user interface approach has be developed. This paper presents a user interface approach, as part of the MAPVI project [1], that allows BVIP access visual, non-verbal and spatial information of business meetings in a user-friendly manner, using only off-the-shelf hardware. Additionally, it presents results of user tests of this novel user interface.

Reinhard Koutny, Klaus Miesenberger

Interactions for Text Input and Alternative Pointing

Frontmatter
Study of User Behavior When Using a List of Predicted Words

In this article, we present a study on the user’s gaze tracking during a text input task on a soft keyboard associated with a word prediction list. The purpose of this study is to investigate user strategies for using the prediction list. When does he watch it? How frequently? Does he look at the whole list? our first results shows that the user only looks at the top of the list. Therefore, it does not use the prediction list optimally and enters more characters on the soft keyboard. Moreover, typing on the soft keyboard is also penalized by the presence of the list next to the keyboard because the user looks at it without necessarily using it. This sub-optimal use of the list coupled with slower input on the soft keyboard therefore results in a slower average input speed on the soft keyboard associated with a list.

Mathieu Raynal, Georges Badr
TBS: Two-Bar Single-Switch Scanning for Target Selection

We present a two-dimensional (2D) pointing technique for motor-impaired users who generally use single-input switch. The technique, called TBS $$^3$$ 3 for “two-bar single-switch scanning”, uses two bars and proceeds in two steps: moving a vertical bar then moving a horizontal bar. The user starts and stops each bar by activating the single input switch. Selection is made at the intersection of the two bars when the horizontal bar is stopped by the user. We present two variations of the technique. In the first (“one-way”), the bars move only in one direction, and return to their origin after each selection. In the second (“button”), the user controls the direction by a selecting an on-screen soft button before bar movements starts. The techniques were evaluated in a experiment with twelve participants using the 2D target-selection task described in ISO 9241-411. Due to the two-stage pointing process, the task was inherently slow with a mean movement time of 8.4 s per trial. The mean error rate was just under 10% with the one-way method (8.0%) more accurate than the button method (11.1%). Throughput was 0.31 bps overall with values of 0.26 bps for the one-way method and 0.36 bps for the button method.

Mathieu Raynal, I. Scott MacKenzie

Open Access

Impact of Using an Eye-Gaze Technology by a Young Adult with Severe Cerebral Palsy Without Speech

This case study explores an eye-gaze technology intervention for a young adult with severe physical and speech difficulties and visual impairments. Data were collected over a six-month intervention period encompassing measures on pupil’s occupational performance of computer activities and psychosocial impact, and interviews with the user, the parents, and the teacher on the technology acceptability. The results showed that the six-month intervention enhanced the pupil’s performance in three computer activities and led to a positive psychosocial impact. The parent and teacher described the intervention as appropriate to increase the pupil’s self-expression and interaction with others, and there was no adverse event during the study period. The pupil demonstrated motivation to use the eye-gaze technology after the intervention continuously. In conclusion, this study shows that a young adult with severe motor impairments and visual problems can benefit from using eye-gaze technology to increase participation in leisure activities, communication, and social interactions.

Yu-Hsin Hsieh, Mats Granlund, Ai-Wen Hwang, Helena Hemmingsson
Proposal of “micro:bit PC” Powered by Numeric Key Programming for both Visually Impaired and Sighted Elementary School Students

In the informational society, elementary school students are expected to learn programming. The BBC micro:bit, which has many sensors, is employed as one of the programming materials. However, MakeCode, which is a block programming editor of the micro:bit, is not accessible for visually impaired yet. Regarding this problem, we have proposed a new and easy programming environment for a mobile robot kit involving the micro:bit. The environment does not require any PC or tablet so visually impaired students can use it. Therefore, visually impaired students were able to obtain the programming skill within 15 min at maximum from their first touch of the robot. In this paper, we explain on the details of the command set of our robot and how to use the sensors of the micro:bit with our programming environment for the robot kit.

Yoshihiko Kimuro, Taishi Takiuchi, Ken’ichi Furusato, Takafumi Ienaga
Extended Mouth/Tongue Gesture Recognition Module for People with Severe Motor Dysfunction

We researched gesture interfaces for people with motor dysfunction who cannot use normal interface switches. For this purpose, we have developed nine gesture recognition modules. One of them is the tongue recognition module that detects whether the tongue is in or out for the switch interface. In the process of collecting data on various types of mouth and tongue gestures and user’s demands, we discovered three useful types of mouth-related gestures: mouth open/close, tongue in/out, and movement around the lips. These gestures are also useful when performed simultaneously. We conducted several experiments using data of people with motor dysfunction and inspected the conditions for real use. Furthermore, we conducted a basic experiment with healthy people to recognize three gestures simultaneously.

Ikushi Yoda, Kazuyuki Itoh, Tsuyoshi Nakayama
Accessible Hands-Free Input Methods for VR Games

New ways to interact with VR environments come with both opportunities and challenges for accessibility. In this work, we summarize different possible hands-free input methods to play a VR game and investigate their applicability. Using a serious game as a Proof of Concept we implemented five focus methods (head-tracking, scanning, head-movement, eye-tracking and voice control) and three activation methods (button, dwelling and blink) exemplary using the Oculus Go standalone VR headset and the Unity game engine. We analyzed the suitability of these methods in a pre-study that excluded head movements for controlling the game based on user feedback. The remaining input methods were evaluated in an explorative user study in terms of operability and ease of use.

Fiona Heilemann, Gottfried Zimmermann, Patrick Münster
Gaze-Contingent Screen Magnification Control: A Preliminary Study

People with low vision often use screen magnification software. Screen magnification requires continuous control of the onscreen content by moving the focus of magnification with the mouse or the trackpad. In this contribution, we explore the possibility of controlling the focus of magnification by means of the user’s own eye gaze, which is measured by a commercial gaze tracker. We conducted two small experimental studies with individuals with impaired central vision, who used two screen magnification modalities to read two different types of documents. In the first study, mouse tracks and gaze point tracks were collected during manual control for later analysis. In the second study, the center of magnification was controlled by the user’s own gaze, using two different control mechanisms. This preliminary study highlights the potentials and shortcomings of gaze-contingent screen magnification control for easier access of onscreen content with low vision.

Roberto Manduchi, Susana Chung
Pardon? An Overview of the Current State and Requirements of Voice User Interfaces for Blind and Visually Impaired Users

People with special needs like blind and visually impaired (BVI) people can particularly benefit from using voice assistants providing spoken information input and output in everyday life. However, it is crucial to understand their needs and include them in developing accessible and useful assistance systems. By conducting an online survey with 146 BVI people, this paper revealed that common voice assistants like Apple’s Siri or Amazon’s Alexa are used by a majority of BVI people and are also considered helpful. In particular, features in audio entertainment, internet access, and everyday life practical things like weather queries, time-related information (e.g., setting an alarm clock), checking calendar entries, and taking notes are particularly often used and appreciated. The participants also indicated that the integration of smart home devices, the optimization of existing functionalities, and voice input are important. Still, also potentially negative aspects such as data privacy and data security are relevant. Therefore, it seems particularly interesting to implement offline data processing as far as possible. Our results contribute to this development by providing an overview of empirically collected requirements for functions and implementation-related aspects.

Christina Oumard, Julian Kreimeier, Timo Götzelmann
Learning a Head-Tracking Pointing Interface

For people with poor upper limb mobility or control, interaction with a computer may be facilitated by adaptive and alternative interfaces. Visual head tracking has proven to be a viable pointing interface, which can be used when use of the mouse or trackpad is challenging. We are interested in new mechanisms to map the user’s head motion to a pointer location on the screen. Towards this goal, we collected a data set of videos of participants as they were moving their head while following the motion of a marker on the screen. This data set could be used to training a machine learning system for pointing interface. We believe that by learning on real examples, this system may provide a more natural and satisfactory interface than current systems based on pre-defined algorithms.

Muratcan Cicek, Roberto Manduchi

Cognitive Disabilities and Accessibility

Frontmatter
Cognitive Disabilities and Accessibility
Introduction to the Special Thematic Session

This STS, following a series of STSs and sessions on Cognitive Disabilities and Accessibility in ICCHP over more than two decades, documents the development on how the neurodivergent people and their support/care sector develops in defining and addressing needs towards R&D for both Accessibility and Assistive Technologies (AT). It reports on progress in guidelines, standards, methods, techniques, and tools for addressing Accessibility. And it gives a comprehensive overview to new and innovative AT solutions and service provision models for better participation, including studies and reflection on efficiency, effectiveness and impact on social inclusion and quality of life. This is accompanied by reflections on socio-economic, business and policy/political developments.

Klaus Miesenberger, Susanne Dirks, Christian Bühler, Peter Heumader
First Attempt to an Easy-to-Read Adaptation of Repetitions in Captions

Subtitles of audiovisual content produced in the same language as the oral discourse are called captions. Such type of subtitles is crucial to ensure that audiovisual resources have inclusive and equal access for people with functional diversity. When talking about people with reading comprehension difficulties, captions must be written in easy reading. During the subtitling process, it is important to bear in mind that the oral mode includes some unique characteristics such as the use of punctual reiterations or redundancies. However, excessive repetition in the written mode slows down reading and makes it difficult to understand. Currently, repetition is not considered as a problematic aspect in the Easy-to-Read (E2R) Methodology, since this linguistic resource is not frequent in the written mode. Despite this, we believe that some features of the oral mode, such as repetitions, should be considered within the captioning process. Hence, our current research is focused on discovering whether captions with repetitions, coming from the oral mode, can be a problem for people with cognitive disabilities. To achieve such a goal, we performed a user study whose main goal is to find out whether people with reading comprehension difficulties prefer audiovisual captions with or without repetitions. Initial findings indicate that captions without repetitions are the most preferred ones. For this reason, we have also created a method for automatically adapting repetitions in captions following an E2R approach.

Mari Carmen Suárez-Figueroa, Isam Diab, Álvaro González, Jesica Rivero-Espinosa
ACCESS+: Designing a Museum Application for People with Intellectual Disabilities

Inclusive solutions are essential to improve the user experience and overall accessibility. They contribute to the independence and participation of people with disabilities and can be designed for a wide variety of contexts. In this paper, we describe a design cycle from ideation to testing and redesign of ACCESS+, an accessible application to navigate through museum content focusing on people with Intellectual Disabilities (ID). We have focused on personalized and inclusive features so that users could tailor to their needs and preferences icons and font sizes, labels, and backgrounds. Also, users could make sense of the text by looking at symbols via Augmentative and Alternative Communication (AAC), and by listening to text-to-speech of full text with highlight, tone, and pitch configuration. Finally, users could provide different forms of feedback: ratings and comments. We conducted heuristic evaluations with an educator and a psychologist, both specialists in inclusive education, redesigning the interface and moving from a system to a user-friendly terminology. We also followed the specialists’ suggestions and made the icons and text of the UI more accessible.

Leandro Soares Guedes, Valentina Ferrari, Marilina Mastrogiuseppe, Stefania Span, Monica Landoni

Open Access

Investigating the Usability of Voice Assistant-Based CBT for Age-Related Depression

To combat the global shortage of mental health services, new solutions - such as computerized therapy options - have to be found. While research in this field has been ongoing for several decades and approaches such as chatbot-based cognitive behavioral therapy (CBT) have already shown to be effective in reducing symptoms of depression for younger adults, voice assistant-based (VA-based) approaches have thus far not been investigated deeply. However, especially for elderly people with depression VA-based systems could yield benefits - such as the avoidance of physical accessibility issues. Therefore, we propose the design of a VA-based system capable of delivering selected methods from CBT to elderly users with depression in order to investigate its usability. To assess the usability of the conceptualized system in comparison to a chatbot-based approach we conducted a randomized controlled A/B testing experiment with 14 participants. Results indicate a good usability and acceptance of the designed system and a preference for the delivery of CBT-methods via voice assistant rather than via chatbot, especially among elderly participants.

Julian Striegl, Marie Gotthardt, Claudia Loitsch, Gerhard Weber
Data-Driven User Profiling and Personalization in Tiimo: Towards Characterizing Time Management Behaviors of Neurodivergent Users of a Scheduling Application

Deficits with time management and other cognitive functions can stem from multiple causes and be found across different diagnostic conditions. At the same time, cognitive function can differ within diagnostic classes, which calls for adaptable and personalized assistance. A great deal of literature on cognitive assistive technology (CAT) focus on diagnostic populations rather than cognitive impairments across different conditions. This study reports the initial steps towards a data-driven approach to map out the characteristics and behavior of users of a time management app, Tiimo, originally targeting children with ADHD. Based on results from a questionnaire and analysis of user activity data, findings indicate a tendency of attracting a more heterogeneous user population compared to the originally intended target group, thus supporting the need for a more complex and data-driven ‘design for all’ approach to CAT rather than delimitations based on diagnostic groups. Preliminary findings from the analysis of activity data across user groups and diagnoses show that users generally schedule fewer than five daily activities and most often in the morning, suggesting a potential emphasis on support particularly during morning routines. However, the analysis also highlights the need for more data points to enable assessment of progress, motivation, and effectiveness of the technology. Next steps include a more detailed analysis of user activity that takes different types of behavior and other relevant factors into account by applying NLP to further develop data-driven approaches to user profiling and personalization in time management apps for neurodivergent users.

Sofie Otto, Brian Bemman, Lykke Brogaard Bertel, Hendrik Knoche, Helene Lassen Nørlem

Open Access

Voice Assistant-Based CBT for Depression in Students: Effects of Empathy-Driven Dialog Management

With a rising number of students with depression, new low-threshold solutions have to be found to strengthen the resilience against and help those affected by mental disorders. One approach lies in the usage of chatbots (CBs) to provide tools based in cognitive behavioral therapy (CBT) that can be used independently in order to reduce symptoms of depression. To ensure the adherence to such systems, a good usability and acceptance is important. Conversational agents (CAs) that provide CBT-based content should further be sensitive to the users emotional state, as empathy is one central aspect of therapy. While promising research has been going on in the field of CB-based empathy-driven CBT, voice assistant-based (VA-based) solutions have thus far not been investigated deeply. Therefore, we propose a VA-based, empathy-driven system, capable of delivering selected methods from CBT to students with depression.To assess the effects of empathy-driven dialog management on perceived usability and acceptance, we conducted a single blind randomized controlled A/B testing experiment with 10 participants. While the application of empathetical dialog management shows no benefits to the usability and acceptance, results overall indicate a good usability and acceptance of the system in the target group.

Marie Gotthardt, Julian Striegl, Claudia Loitsch, Gerhard Weber
Augmented Reality Game for Children with Autism Spectrum Disorder

Games play an important role in the development of children. Especially when it comes to children with special needs like autism, it is important to have a different approach when it comes to learning. Shapes and colours are the most important fundamental skills that help in the recognition of objects around and help with letters and words. Balloon Pop is an application that is designed keeping in mind the requirements of children diagnosed with autism. The paper gives an idea of the user’s needs and requirements and shows how the project solves the problem with augmented reality game design. The goal of the project is to design a game for autistic children that will help them learn the abstract concept of shapes and colours. The Digital game-based learning methodology (DGBL) is used in developing the game. The paper consists of the design rules that have been followed in designing for better interaction along with hardware and software architecture describing the flow of the project. Also, at last, the test and evaluation and 8 users have been done by using the System Usability Scale (SUS) tool. The application has been proven to be helpful for children along with parents/therapists to achieve the goal to make children interact and communicate freely, increase focus, and learn the abstract concepts of shapes and colour easily.

Mario Covarrubias Rodriguez, Shefali Mehta, Milton Carlos Elias-Espinosa

Open Access

Making Person-Centred Health Care Beneficial for People with Mild Cognitive Impairment (MCI) or Mild Dementia – Results of Interviews with Patients and Their Informal Caregivers

In the health care sector, person-centred treatment approaches have shown the potential to improve treatment outcomes and quality of life of patients. In particular, this applies where patients are living with complex conditions like multimorbid older patients with Mild Cognitive Impairment (MCI) or mild dementia. Such treatment approaches quite often include input from modern health technologies like health/home monitoring platforms which also offer services to patients for self-management of their conditions. This approach is also followed in the research project CAREPATH (An Integrated Solution for Sustainable Care for Multimorbid Patients with Dementia). To achieve acceptance of such complex health technologies, their services must be beneficial in the eyes of target end users which included in the case of CAREPATH, the patient’s informal caregivers. Therefore, understanding the user requirements of patients and their informal caregivers is of utmost importance which was achieved in CAREPATH by interviews. These revealed that patients’ preferences in regard to what services and information shall be provided to them shall be limited to what they deem necessary which is highly personal. Informal caregivers as opposed to patients, are much interested in receiving most possible information about their care-dependent’s health status. Thus, provision of services and information for these user groups need to be highly customizable to their personal preferences and needs.

Henrike Gappa, Yehya Mohamad, Martin Breidenbach, Pedro Abizanda, Wolfgang Schmidt-Barzynski, Antje Steinhoff, Timothy Robbins, Harpal Randeva, Ioannis Kyrou, Oana Cramariuc, Cristiana Ciobanu, Theodoros N. Arvanitis, Sarah N. Lim Choi Keung, Gokce Banu Laleci Ertürkmen, Mert Gencturk, Mustafa Yüksel, Jaouhar Ayadi, Luca Gilardi, Angelo Consoli, Lionello Ferrazzini, Carlos A. Velasco

Augmentative and Alternative Communication (AAC): Emerging Trends, Opportunities and Innovations

Frontmatter
Augmentative and Alternative Communication Emerging Trends, Opportunities and Innovations
Introduction to the Special Thematic Session

Augmentative and Alternative Communication (AAC) technologies, training and support have benefitted from significant emerging trends in recent years to cope with the changing environments experienced by many users. The landscape of inclusion, whether it means different forms of digital accessibility, tele-support or more built-in assistive technology in everyday devices, has helped many more individuals with speech and language difficulties communicate effectively. There has also been an increased use of artificial intelligence, including machine learning and natural language processing with improved multilingual automatic speech recognition (ASR) and text to speech, location capturing apps and the Internet of Things being just a few of the technologies providing the world of AAC with a wealth of exciting emerging trends, opportunities and innovations. This special thematic session aims to provide an insight into some of the trends developing across Europe and the wider community.

E. A. Draffan, David Banes
Open Licensed AAC in a Collaborative Ecosystem

A collaborative ecosystem that encompasses the use of open-licensed augmentative and alternative communication (AAC) solutions and systems has the potential to provide positive outcomes for children with severe speech and language difficulties. This has been shown through a project that highlighted the willingness to provide a considerable amount of teamwork and participation of families and carers involving 124 children with complex communication needs across three Eastern European countries. Participation was based around a UNICEF hub in each capital city. The hub provided support for small groups of AAC experts sharing their knowledge with limited resources and widely varying groups of other professionals, families and carers of potential AAC users. Initial face to face training sessions provided introductory sessions to open licensed AAC solutions and systems. These sessions were backed up by ‘anytime’ access to an open licensed eLearning platform containing freely adaptable interactive AAC online training resources to be translated into modules by participants in each country. The level of content was based on the first three levels of the European Qualifications Framework. There followed the development of pictographic symbol sets to enhance the localization of already available sets suitable for children. Cultural, linguistic, and social settings were catered for within an open-source communication app. Feedback and formal evaluations provided by parents, carers and professionals showed that the benefits to the children were not just in AAC use, but also in their social competency levels, self-esteem and adaptability.

E. A. Draffan, David Banes
Development of a Location and Situation Based Augmentative and Alternative Communication Service

Mobile Augmentative and Alternative Communication (AAC) applications have great advantages in portability. Choosing the appropriate AAC board for a user’s current location or situation is a very difficult and hard job for people with cerebral palsy, motor disabilities, or intellectual disabilities due to their limited motor skills or cognitive abilities. We developed a location and situation-based AAC service that automatically recommends a board suitable for the AAC users’ current location and communication situation. Through a survey to identify places that those people with disabilities and their facilitators consider important, 20 built-in place type AAC boards were developed. The users can also create their own customized AAC boards for the specific locations where they visit frequently. The built-in and customized AAC boards for locations may consist of situation sub-boards with the order of communication contexts in the locations. We developed an algorithm to recommend the most appropriate location and situation-based AAC board from the built-in and customized boards based on the user’s current location and communication context in that location. The location and situation-based AAC service is expected to improve the convenience of AAC users who have difficulties in using AAC applications due to their limited motor skills or cognitive abilities, and to facilitate the independent life of those AAC users.

Seo-Yeong Ahn, Ki-Hyung Hong, Kyungyang Kim, Heeyeon Lee
Augmentative and Alternative Interaction Service with AI Speakers to Access Home IoT Devices and Internet Services for People with Multiple Disabilities

With the growing demand for Internet-of-Things (IoT) and internet services for people with multiple disabilities, including communication and motor disabilities, we propose an Augmentative and Alternative Interaction (AAI) service for them. Interacting with IoT devices and internet services can be done by voice-controllable smart (or Artificial Intelligence (AI)) speakers. AI speakers are difficult to use for those people with multiple disabilities. The direct application of Augmentative and Alternative Communication (AAC) to the interaction is ineffective, since AAC mainly focuses on face-to-face communication and the sound produced by Text-to-Speech (TTS) for AAC symbols does not reflect the AI speakers’ speech recognition features. Based on a user survey for AI speaker commands, we developed AAI symbols and boards to interact with home IoT devices and internet services through AI speakers. In order to improve speech recognition performance of AI speakers, we developed the TTS production format for AAI service, including wake-up words, silence intervals and commands. The AAI service allows people with multiple disabilities to control home IoT devices by themselves and to enjoy their spare time on their own by interacting with internet services such as music streaming, Over-the-Top (OTT) media services, and news searching through AI speakers.

Se-Hui Ryu, Ki-Hyung Hong, Soojung Chae, Seok-Jeong Yeon

Language Accessibility for the Deaf and Hard-Of Hearing

Frontmatter
Language Accessibility for the Deaf and Hard-of-Hearing
Introduction to the Special Thematic Session

The Special Thematic Session on Language Accessibility for the Deaf and Hard-of-Hearing focuses on two separate groups of individuals – the hard-of-hearing and the deaf. Both groups have faced barriers to language access, but they have different perspectives and priorities. Those who are hard-of-hearing still use a spoken language as their preferred language even though they have experienced a hearing loss. Those who identify as deaf often do not use a spoken language but a sign language as their preferred language. This session contains a wide range of papers that illustrate different aspects of using language technologies in various environments. Studies show the possibilities for improvements in higher education, captioning, and sign language interpretation, all with the common purpose of fostering better access to language.

Matjaž Debevc, Rosalee Wolfe, Sarah Ebling
Contemporary Assistive Technologies for Students with Hearing Loss in Higher Education

The COVID-19 pandemic has brought several challenges to students with hearing loss in Higher Education. Accordingly, assistive, mobile and other contemporary accessible technologies have become more important than ever before. This study provides an overview of the needs and requirements of students with hearing loss in Higher Education. The findings, firstly, include a list of assistive technologies for the purpose of communicating, listening to speech and audio devices, sound recognition, informing, sensing, alerting and learning. Secondly, the findings provide a list of mobile technologies for the purpose of communicating, managing sound, sensing, alerting, learning sign language, learning in general and leisure activities. Thirdly, the findings encompass new dimensions of contemporary technologies, having the potential to support students with hearing loss in challenging educational settings. These findings may primarily serve students with hearing loss, practitioners in Higher Education and other relevant stakeholders.

Ines Kožuh, Peter Čakš, Matjaž Debevc
Comparing the Accuracy of ACE and WER Caption Metrics When Applied to Live Television Captioning

The development of caption metrics is relatively new in the accessibility research community. However, little work has been done comparing the effectiveness of newly developed caption metrics. More specifically, in low accuracy settings such as live television, where users report the most difficulty using captions. Through a user study with fifteen participants, we compared two caption metrics systems, Word Error Rate (WER) and Automated-Caption Evaluation (ACE), for their accuracy in evaluating caption quality in live television. We compared human-perceived quality statistics with each caption metric’s data. Analysis of the correlation between human statistics and each caption metric found that WER had a slightly higher correlation with participants. We found that ACE was more sensitive to errors that WER, and penalized captions more than participants. However, the difference in performance between WER and ACE was not statistically significant, and neither WER nor ACE are optimized for use with live television captioning. Future work should explore how caption metrics could be better optimized for use with live television.

Tian Wells, Dylan Christoffels, Christian Vogler, Raja Kushalnagar
Workload Evaluations for Closed Captioners

Live captioning in broadcasting involves the translation from spoken words in broadcasted programming to text equivalents, speaker identification and some non-speech audio information in real-time where there is little or no opportunity for editing, or correction. Some of this live content can be too fast to type, and/or read, which then can translate into difficulties and barriers for viewers. Recently, paraphrasing has been permitted in order to attempt to mitigate some of these difficulties, but different cognitive elements must be recruited compared with verbatim translation. This research presents preliminary results of live captioner’s subjective mental workload (SMW) using the NASA TLX, and their experiences with paraphrasing. We hypothesize that the cognitive processes of paraphrasing increase SMW. Results indicate that live captioners experience high SMW particularly for fast-paced sports. Paraphrasing may contribute to higher perceived SMW for live captioning fast-paced sports, talk shows and weather.

Maria Karam, Christie Christelis, Evan Hibbard, Jenny Leung, Tatyana Kumarasamy, Margot Whitfield, Deborah I. Fels
Caption User Interface Accessibility in WebRTC

We investigate caption information and communication preferences by deaf and hard-of-hearing users of captions or subtitles, which are the text representation of language, whether spoken or written, in an open-source teleconferencing platform: WebRTC. We find that users prefer captions that are displayed next to the speaker’s head and a texting setting that allows real-time text communication in a letter-by-letter format for most teleconference settings.

Michelle Olson, Ianip Sit, Norman Williams, Christian Vogler, Raja Kushalnagar

Open Access

See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display

Access to audible information for deaf and hard-of-hearing (DHH) people is an essential component as we move towards a diverse society. Real-time captioning is a technology with great potential to help the lives of DHH people, and various applications utilizing mobile devices have been developed. These technologies can improve the daily lives of DHH people and can considerably change the value of audio content provided in public facilities such as museums. We developed a real-time captioning system called See-Through Captions that displays subtitles on a transparent display and conducted a demonstration experiment to apply this system to a guided tour in a museum. Eleven DHH people participated in this demonstration experiment, and through questionnaires and interviews, we explored the possibility of utilizing the transparent subtitle system in a guided tour at the museum.

Ippei Suzuki, Kenta Yamamoto, Akihisa Shitara, Ryosuke Hyakuta, Ryo Iijima, Yoichi Ochiai
Backmatter
Metadaten
Titel
Computers Helping People with Special Needs
herausgegeben von
Klaus Miesenberger
Georgios Kouroupetroglou
Katerina Mavrou
Roberto Manduchi
Dr. Mario Covarrubias Rodriguez
Petr Penáz
Copyright-Jahr
2022
Electronic ISBN
978-3-031-08648-9
Print ISBN
978-3-031-08647-2
DOI
https://doi.org/10.1007/978-3-031-08648-9

Neuer Inhalt