Multiexperience

The development of new devices, such as smart speakers or wearables, and recent advances in artificial intelligence (AI) that facilitate more natural interactions via speech or gestures are changing the interplay between user, task, and technology within information systems (IS). Today, people own multiple different devices, such as personal computers, smartphones, tablets, or smart speakers, and use them interchangeably, often switching between multiple devices in order to complete a task (Levin 2014; Westcott et al. 2020). In addition, many of these devices afford users new ways of interacting with them through touch, speech, or gestures (Turk 2014). For example, customers can shop at Amazon using multiple devices and multiple interaction modalities as well as combinations thereof (e.g., Amazon’s Echo Show devices combine speech interaction with a touchscreen display). The same trend can be found at the workplace, where employees can, for example, use natural language to interact with enterprise resource planning (ERP) systems (e.g., SAP CoPilot) or business intelligence and analytics (BI&A) systems (e.g., Tableau Ask Data).

This trend has not gone unnoticed by market research firms that recently introduced the term multiexperience (MUX) as a key area of strategic importance in the next years (Gartner 2019). There also is a plethora of existing research related to MUX that provides insights into how users interact with IS across different devices and modalities and how to design for MUX (Brudy et al. 2019; Li and Zhang 2005; Turk 2014; Zhang et al. 2009). However, what is new today is that the sheer number of available devices and mature modalities presents an opportunity – and challenge – to better meet users’ needs and preferences when they interact with an IS to perform tasks. Similar tasks can be performed quite differently and may result in different outcomes, depending on the devices used (e.g., smartphones vs. smart speaker) and modalities available (e.g., clicking on a screen vs. speech input) (Diederich et al. 2020; Rzepka et al. 2020a). Thus, there is a need to improve our understanding of the nature of MUX and the roles of devices and modalities within IS. Considering that many devices (e.g., virtual reality headsets) and modalities (e.g., speech interaction) are now beginning to reach a level of maturity which allows widespread application, we believe the time is ripe to (re)define the concept of MUX.

The goal of this catchword is to build a bridge between the new term of MUX and existing research in our field in order to provide a solid conceptual grounding for MUX and identify future research opportunities for the BISE community. Drawing on the rich body of research on multi-device and multimodal IS in the fields of IS and human–computer interaction (HCI), we propose a clear conceptualization of MUX and provide a framework of three guiding paths toward MUX that may be equally useful for researchers and practitioners in the BISE community. Additionally, we describe several real-world examples to explain the benefits and challenges of each path in our framework and offer practical guidance for moving along these paths toward MUX. Finally, we outline promising areas for future research and highlight the unique position of the BISE community in capitalizing on these opportunities.

In the early days of personal computing, researchers and practitioners primarily focused their efforts on a single device – the personal computer (PC). However, already in 1991, Mark Weiser shared his vision of a world in which people can interact with content across multiple computing devices in different shapes and sizes (Weiser 1991). This idea served as inspiration for research that went beyond a single user at a single computer (Brudy et al. 2019). A classic example is Rekimoto’s seminal work from the late 1990s on interaction techniques that crossed device boundaries of multiple portable computers and displays on table/wall surfaces (Rekimoto 1997; Rekimoto and Saitoh 1999). Around the same time, commercial products such as the first BlackBerry device were introduced that offered new ways to do everyday tasks that would typically be done on a computer in an office (Lyytinen and Yoo 2002). For example, people could not only send and receive emails but also synchronize their schedule, tasks, and contacts with their PC – something that is commonplace nowadays but represented a major leap forward at that time. Since the advent of mobile devices in the 1990s in general, and smartphones in the 2000s in particular, the number and diversity of devices has grown significantly. In addition, many companies allow employees to use private devices for work purposes, and vice versa (Köffer et al. 2015). Today, the most popular devices range from PCs, smartphones, tablets, and TVs to smart speakers, smartwatches and augmented (AR) or virtual reality (VR) devices (GlobalWebIndex 2020). Each device is characterized by certain display capabilities (e.g., screen size), processing power, input/output modalities, and sensors (Levin 2014). Furthermore, people use devices much differently than they did 10 or 20 years ago. Since many tasks span multiple devices (Dearman and Pierce 2008), people use several devices simultaneously and switch between them to complete a single task (Brudy et al. 2019). For example, Netflix’s “Continue Watching” feature allows customers to start watching a movie on one device (e.g., a TV in the living room) and continue watching on another device (e.g., a smartphone or tablet) while commuting or waiting for an appointment (Netflix 2013). However, there are also several technical challenges associated with multi-device IS, particularly when it comes to sharing information and keeping it consistent across multiple devices (Dong et al. 2016). Commercial solutions, for example, are often limited to devices within a particular manufacturer’s ecosystem (e.g., Apple) and there are only few open standards that support the integration of multiple devices (Brudy et al. 2019). Nonetheless, current trends suggest that researchers and practitioners cannot consider the PC, smartphone, or any other device as a standalone platform anymore, but need to understand and design for use patterns across multiple devices (Levin 2014).

An important distinguishing feature of the aforementioned devices is that they offer users a wide variety and diversity of interaction modalities (hereafter referred to as modalities for simplicity). Modality broadly refers to the type of communication channel used to convey or acquire information (Nigay and Coutaz 1993). This includes both input modalities (i.e., users providing data to the system) and output modalities (i.e., users receiving data from the system). Users provide input using their effectors (e.g., limbs, eyes, vocal system, head) and perceive output through their five senses (i.e., sight, hearing, touch, smell, and taste). For example, in the interaction with a PC, users primarily provide input through typing on a keyboard or mouse clicks using their fingers (i.e., limbs). Further input modalities, such as speech, mid-air gestures, and eye gaze, that leverage other effectors are possible but less common today. In terms of output modalities, users primarily rely on their visual sense since most applications on a PC feature a graphical user interface. Secondary output modalities that leverage other senses, such as audio output (e.g., “beeping” when an error occurs), may also play a role. Table 1 provides an overview of different categories of devices and their input as well as output modalities.

Although humans employ multiple senses and effectors to interact with the world around them, research in the fields of IS and HCI has historically been focused on unimodal interaction (i.e., using only a single input and a single output modality) (Liu et al. 2019; Turk 2014). An example is the PC that displays text on a screen with a keyboard for input. However, advances within the AI subfields of natural language processing and computer vision as well as affordable sensor technology are paving the way toward more natural interactions that replace or complement traditional modalities (Turk 2014). Multimodality refers to the use of more than one input and/or output modality in the interaction (Nigay and Coutaz 1993). These modalities can be used simultaneously or sequentially during the interaction. For example, in the interaction with an Amazon Echo Show device, users can provide input via speech (e.g., “Alexa, what’s the weather like in Berlin today?”) and its touch screen (e.g., touching a button) and receive spoken output (e.g., “The current weather in Berlin is …”) and visual output on the screen (e.g., a weather forecast). The key assumption is that “well-designed multimodal systems integrate complementary modalities to yield a highly synergistic blend in which the strengths of each mode are capitalized upon and used to overcome weaknesses in the other” (Oviatt 1999, p. 74). Indeed, research has shown that multimodality can lead to higher task performance (Lee et al. 2001) and improve learning outcomes (Suh and Lee 2005). Similar to multiple devices, the integration of input from multiple modalities is a key technical challenge (Reeves et al. 2004). The main reason is that due to the unique characteristics of each modality, there are no obvious points of similarity and therefore no straightforward ways to connect them (Turk 2014). Over the years, several so-called fusion approaches have been developed to address this challenge (Jaimes and Sebe 2007). In general, fusion can be performed at different levels, ranging from feature level (i.e., integrating input signals) to higher semantic levels (i.e., integrating common meaning representations derived from different modalities) (for an overview, see Jaimes and Sebe 2007). Although significant progress has been made, technical challenges related to the integration of modalities remain and the development of fusion approaches continues to be an active area of research.

To summarize the conceptual foundations of MUX, Table 2 provides an overview of the two research streams on multi-device and multimodal IS with key papers and exemplary artifacts.

Against the backdrop of the conceptual foundations, we define MUX as the user’s perceptions and responses resulting from the use and/or anticipated use of an IS that leverages multiple devices and/or multiple modalities. As such, it combines the terms multi (i.e., multi-device, multimodality) and (user) experience¹ to account for the increasing prevalence of more than a single device and/or modality within an IS. New devices and advanced interaction modalities have certainly made scenarios a reality that belonged in the realm of science fiction just a few years ago (e.g., talking to an ERP system or wearing smart glasses that display information directly in the field-of-view). However, since both the IS and HCI field have a long tradition of investigating how users interact with IS across different devices and modalities and how to design IS with multiple devices and/or modalities, we argue that MUX is not an entirely new phenomenon. What is new is that the sheer number of available devices and modalities today provides a greater opportunity and challenge to better meet users’ needs and preferences when they interact with an IS. In addition, many devices (e.g., virtual reality headsets) and modalities (e.g., speech interaction) are now beginning to reach a level of maturity that allows widespread application.

To shed a more nuanced light on the concept of MUX, we propose and describe a conceptual framework of guiding paths toward MUX. As depicted in Fig. 1, the framework conceptualizes MUX based on the two axes of devices and modalities, illustrating the shift from single to multiple devices and/or modalities. Drawing on previous research on multi-device and multimodal IS, we propose three paths toward MUX that differ not only in their reliance on multiple devices and/or multiple modalities, but also in their prevalence in prior literature. They are intended to serve as starting points for individuals and organizations who seek to proceed on the path toward MUX. In this spirit, our framework is meant to assist; not to constrain or suggest that other paths are not possible. There are four important points to be highlighted. First, our framework suggests that MUX is not achievable when there is only a single device and a single modality. Fundamentally, MUX requires at least two devices or two modalities, but not necessarily both (i.e., multiple devices and multiple modalities). Although many of today’s devices technically support multiple modalities, applications running on these devices do not always capitalize on this potential. For example, a smartphone-only application that only supports touch input and visual output would not be able to achieve MUX. Second, our framework allows for variance in MUX. Similar to the use of the UX concept, MUX can vary from low to high. Just because another device is supported or another modality is added does not automatically imply that MUX has improved. For example, most websites today can be accessed from multiple devices, but a website that offers the exact same layout and content across all devices would achieve rather low MUX (e.g., because text could be difficult to read on a smartphone or large images could result in slow loading times). In contrast, higher MUX could be achieved when the website is optimized for each device (i.e., using a responsive design) or dedicated apps are developed for smartphones and tablets. Consequently, the extent to which MUX is achieved also depends on how devices and/or modalities are integrated and allow users to transition from one device or modality to another during use. Third, there are different entry points to each of the three paths. In this sense, our framework is not bound to a specific sequence or set of devices and/or modalities to achieve higher MUX. For example, one company might enter the path to MUX by adding a mobile app to run alongside their website, whereas another might choose a very different path by adding speech interaction capabilities to their ERP system. Finally, our framework suggests that MUX can be achieved by moving along one axis – either vertically along Path 1 (devices) or horizontally along Path 2 (modalities) – or along both axis simultaneously. However, as we explain in the next sections, the greatest potential lies in Path 3 that leverages both multiple devices and multiple modalities rather than focusing on one dimension alone.

This catchword sheds light on the concept of MUX and proposes a framework with two dimensions – devices and modalities – and three different guiding paths toward MUX. Drawing on illustrative examples for each path, we embed the concept of MUX into existing streams of research in IS and HCI, explain benefits and challenges of each path, and offer practical guidance for moving along these paths toward MUX. While substantial research has been conducted on the first two paths toward MUX, less attention has been paid to the third path, which seeks to leverage the combination of multiple devices and multiple modalities. Therefore, many promising research questions remain and the BISE community is well suited to address them. In the following, we suggest four areas for future research on MUX. Table 3 summarizes the identified future research directions and provides illustrative research questions.

First, while this catchword represents a valuable step toward a better understanding of MUX, our conceptualization could benefit from further refinement. Our framework of paths toward MUX provides sufficient structure to guide future research, but also leaves ample room for further exploration of the nature of MUX and the interplay between devices and modalities. For example, future research could systematically identify and classify the many different combinations of devices and modalities that can be used for MUX (e.g., in the form of taxonomies or morphological boxes). Another vital step would be to operationalize the concept of MUX and develop suitable measurement instruments. These instruments would be equally useful for researchers who seek to empirically evaluate MUX and practitioners who want to assess their software products’ utility. Existing measurement instruments for usability and UX, such as the system usability scale (Brooke 1996) and the user experience questionnaire (Laugwitz et al. 2008), could serve as a suitable starting point. Additionally, future research could identify ways to measure MUX objectively using behavioral data, such as interaction logs across devices and modalities, to complement self-report measures.

Second, another promising direction for future research is the empirical investigation of MUX. From an individual perspective, such work could, for example, examine whether and how MUX influences the adoption and use of IS. Since MUX involves utilitarian and hedonic aspects, studies should investigate both instrumental (e.g., better performance) and experiential outcomes (e.g., enjoyment) as well as potential trade-offs and synergies between them. Given the important role of contextual factors in MUX, future research is also needed to better understand how users behave in different contexts (e.g., at work, at home, while riding on a subway), when performing different tasks (e.g., information search, online transactions, entertainment), and when switching between different contexts and tasks. Moreover, future studies should consider how individual differences, such as demographics, personality characteristics, and preferences for different devices or modalities, affect MUX over time and whether users can be classified into different MUX user types (e.g., mobile-first users, keyboard- or touch-only users). Finally, from an organizational perspective, empirical investigations could attempt to shed light on whether and when investments into MUX (e.g., developing an AR app) pay off and how organizations can find and achieve an “optimal” level of MUX.

Third, numerous opportunities exist for future research to deliver new design knowledge through building and/or evaluating innovative MUX artifacts. Such research could follow a design-oriented behavioral research approach (Maedche et al. 2021) to observe and analyze existing MUX artifacts (e.g., SAP CoPilot, Tableau Ask Data, Amazon’s Echo devices) or a design science research (DSR) approach to build new MUX artifacts that tackle important real-world problems. Moreover, a better understanding of whether and how existing design knowledge for single-device and unimodal artifacts can be reused for the design of MUX artifacts would be beneficial. Of particular interest could be to provide design knowledge on how to effectively combine devices and modalities to be able to adapt the MUX to individual users and their changing needs over time. For example, future research could investigate the design of IS that automatically change or recommend modalities and devices according to the users’ current needs. Research in this area may also profit from setting up collaborations with researchers from other fields such as computer science.

Finally, future research should aim to provide methods and tools that support the development, implementation, and management of MUX. While existing methods and tools from areas, such as human-centered design, UX, and software engineering, may be used as a starting point, it is evident that handling the complexity of MUX – resulting from the large number of possible combinations of devices and modalities – requires a new set of methods and tools. For example, future research could develop methodological guidance to help researchers and practitioners choose among the plethora of options in order to identify the most suitable and promising combinations of devices and modalities for a particular purpose. Similarly, methodological guidance on implementing MUX in an existing IT landscape would be valuable. Finally, while many software vendors offer their own MUX development platforms (Gartner 2021), future research could provide platform-independent tools that empower everyone to design for MUX, regardless of whether they use commercial software packages or their own software stack.

Overall, we believe that this catchword offers a fresh perspective on MUX and opens up manifold opportunities for future research. MUX has not only been an important theme in prior IS research, but current trends and the constant technological advancement also indicate that it will continue to be so for the foreseeable future. At the same time, it is clear that the multitude of ways in which devices and modalities can be combined adds another layer of complexity to understanding the interplay between user, task, and technology. For example, it is difficult enough to design a unimodal artifact for one specific device or to rigorously examine how users interact with an IS on a single device using one modality. Going forward, there is little doubt that these difficulties will increase as the number of available devices and mature modalities continues to grow. Given the background, interests, and skills of BISE researchers, we are convinced that the BISE community is well positioned to both address the challenges and take advantage of the opportunities for future research on MUX, and we invite fellow researchers to contribute to this exciting research stream.