Design and Evaluation of a Conversational Agent for Facilitating Idea Generation in Organizational Innovation Processes

Organizations face challenges in discovering and developing innovations due to limited internal resources (Hansen & Pries-Heje, 2017) and the fact that “when focusing on a limited solution space, companies only apply the most obvious instead of the most efficient of all solutions in order to solve an innovation problem” (Lüttgens et al, 2014, p. 342). In this regard, open innovation approaches have been identified to be an effective strategy to improve the efficacy of organizations’ innovation capabilities (Chesbrough, 2003; Lüttgens et al., 2014). Digital platforms, e.g. idea platforms, enable organizations to apply idea sourcing by involving external contributors to access widely dispersed external knowledge and expertise beyond their boundaries (Boudreau & Lakhani, 2013; Cricelli et al., 2021; Di Gangi & Wasko, 2009). However, organizations struggle to harness the potential of idea platforms (Piezunka & Dahlander, 2015), as such idea sourcing initiatives generate highly diverse input whose utilization and valorization remains a key challenge. In particular, the large quantity of contributions pose major challenges in terms of textually unstructured ideas with an insufficient level of detail and indistinct causalities (Barbier et al., 2012; Kipp et al., 2013). As a result, organizations invest a great expenditure of human capacity and time during idea selection to organize and evaluate ideas to select those with high potential (Blohm et al., 2013; Kittur et al., 2013; Merz, 2018). Nevertheless, familiar contributions or ideas with detailed information but little implementation potential might be selected over those with a lack of details and great potential (Bansemir & Neyer, 2009; Piezunka & Dahlander, 2015).

Idea selection could be more efficient, if ideas followed a defined structure to create a common basis to compare them with each other and if they delivered a rich description to establish causalities. One possible way to reach this objective is facilitating external contributors’ idea generation process on idea platforms (Briggs et al., 1998; Dennis et al., 1990; Fjermestad, 2000). Previous research has shown that structured facilitation by a human leads to favorable results for collaborative work practices in small teams (Bittner & Leimeister, 2014; Niederman et al., 1996). However, human facilitation reaches its boundaries for large-scale, distributed idea generation on idea platforms as humans can hardly deal with many different parallel work streams and are not constantly available in asynchronous collaboration settings. With the rise of the so-called “Facilitator-in-a-Box” paradigm (Briggs et al., 2013), an approach has been established to shift facilitation tasks from humans to system restrictions and prompts implemented in automated scripts. However, the implementation of such concepts for idea generation runs the risk of discouraging contributors. More specifically, filling various fields in a standard submission form might reduce contributors’ enjoyment and cognitive involvement, as they usually do not receive direct rewards via the idea platform (Bretschneider, 2012). Therefore, the user interface and process flow should be designed in such a way that they are engaging for contributors (Attfield et al., 2011) to increase the likelihood of continuing participation while diminishing the detrimental effect of declining motivation levels (Corney et al., 2009; Kim et al., 2013). As idea contributors participate voluntarily, it is therefore paramount to ensure an engaging idea generation process to counteract these adverse effects.

Previous studies have shown increased perceptions of social presence on web-based platforms with virtually embedded social cues (e.g., emotionally rich text, personalized greetings) that approximate face-to-face interactions (e.g., Cyr et al., 2007). In addition, several studies have demonstrated that conversational user interfaces can be rich in social cues (e.g., Pütten et al., 2010). Therefore, contributors’ level of engagement and willingness to invest cognitive effort during idea generation could be fostered by the deployment of automated conversation-based facilitation (Schuetzler et al., 2020). To leverage this conversation-based logic, the design of artificial intelligence (AI) involving machine learning (ML) to process natural language can provide an increased level of reactivity and proactivity in comparison to pre-defined time-based sequences of system prompts and state changes. However, the interaction between humans and AI requires more than intelligent algorithms in order to solve specific problems collaboratively and effectively (Harper, 2019; Seeber et al., 2020). In this vein, scholars have recently pointed out that AI-based agents, i.e., in the form of conversational agents (CAs), can be designed to serve the role of a facilitator to support individuals during task execution (Bittner et al., 2019a; Seeber et al., 2018). Moreover, initial research has shown that CAs can guide contributors on idea platforms to generate and submit their ideas in task-oriented conversations (Tavanapour & Bittner, 2018). However, prescriptive design knowledge on how to develop such a solution is still scarce (Bittner et al., 2019b; Diederich & Brendel, 2019; Seeber et al., 2018). Therefore, the following research question is addressed:

Consequently, the aim of this study is to enhance organizations’ idea generation via external contributors with a CA as a facilitator and to lay the foundation for improved subsequent organizational idea selection. Therefore. the AI-based facilitation on idea platforms should result in an engaging process to support individuals in voluntarily generating a contribution to an “open call” (Chesbrough & Brunswicker, 2014; Lüttgens et al., 2014) and yield idea submissions with a common structure comprising specific and detailed descriptions. To investigate the potential of the proposed AI-based facilitator for idea generation on idea platforms, the CA concept needs to be instantiated with a software prototype. Thereby, the implementability of the derived design knowledge can be tested with state-of-the-art CA technology. Furthermore, potential effects of facilitation support by CAs during the idea generation process can be explored. Accordingly, in this study, we present a multi-cycle design science research (DSR) project that addresses the stated challenges and research gap with the following structure. First, we present related work about facilitation of idea generation on idea platforms and CAs as facilitators. Second, we outline the research approach by delineating the steps of the DSR project. Third, derived design requirements (DR) and design principles (DP) are described followed by an instantiation of the CA design with a full-featured CA incorporating insights from previous DSR steps. Subsequently, we present the results of the ex-ante and ex-post evaluation stages. Last, we discuss the findings of the study, its limitations, and present an outlook before closing with a conclusion.

In order to address the research aim of assisting and engaging contributors during idea generation to lay the foundation for a systematized selection of submitted ideas, we conduct a DSR project with multiple consecutive design cycles (see Fig. 1) (Gregor & Hevner, 2013; Vom Brocke et al., 2020). With the design and development of an artifact in the form of a full-featured CA facilitator incorporating insights from previous design cycles, we intend to provide a novel and innovative solution to the prevalent real-world problem of unsystematized and insufficiently engaging idea generation processes that are commonly deployed for open innovation initiatives. To ensure research rigor and generate substantial prescriptive design knowledge, we follow the established iterative six-step approach by Peffers et al. (2007).

Two preceding design cycles were completed to iteratively approach the identified problem. The scope of the first cycle was to gain exploratory knowledge about automated facilitation for idea generation with a CA. Correspondingly, micro and macro scripts were defined to generate tentative design knowledge in the form of interaction scripts (Gregor & Hevner, 2013). The macro script serves to define the process sequence and conversation flow, whereas the micro script specifies relationship-related aspects (e.g., affirmative statements, motivational explanations) for the CA facilitation. To assess the potential of the interaction design for a CA facilitated idea generation process, a Wizard-of-Oz (WoO) experiment was performed (Kelley, 1983). For this purpose, uninformed participants interacted with an undisclosed human wizard, who used the micro and macro scripts to facilitate the idea generation process. The wizard controlled the system to make the participants believe that they are interacting with a CA. The results of the WoO experiment, on the one hand, served as a proof-of-concept for follow-up investigations. On the other hand, the findings were used to inform the improvement of the conceptual CA design. The first cycle was completed by communicating the derived insights (Bittner et al., 2019a).

Guided by the validated micro and macro scripts, an initial CA prototype was developed for automated facilitation on idea platforms. Based on the conversation protocols from the WoO experiment, data was derived to train the open-source NLP framework Rasa¹ for the CA prototype. For the design, DRs were identified with a comprehensive literature search according to Webster and Watson (2002) drawing on justificatory knowledge from the fields of AI, NLP and ML. The prototypical CA was evaluated with a user test. The communication of initial design knowledge and evaluation results completed the second cycle (Tavanapour & Bittner, 2018).

The third cycle represents the focus of this publication. The objective is to combine and extend insights from the first two cycles to address the joint problem identification of this DSR project. Step one (problem identification) has been addressed in the introduction and related work section. In the second step objectives of a solution, previous tentative prescriptive knowledge is expanded by developing DPs, which are based on extended and refined preliminary DRs from cycle two. This revision builds on a literature-based derivation of requirements and an analysis of results from the evaluation in the preceding cycle. In the third step design and development, the DPs are instantiated. Informed by derived DRs a revised and full-featured version of the CA facilitator is implemented. To this end, training data from cycle two is updated with refined intents and entities to improve the performance of the NLP module of Rasa. Moreover, micro and macro scripts from cycle two were utilized to construct the facilitation process sequence of the instantiated CA. Regarding steps four and five, an evaluation of the design comprising ex-ante (demonstration) and ex-post stages (evaluation) is conducted (Venable et al., 2016) (see Table 1). With the ex-ante evaluation, the applicability, operationality, and completeness of the designed artifact for the described problem statement of the DSR project is demonstrated (Sonnenberg & Vom Brocke, 2012). In this evaluation activity, exploratory focus groups (EFG) were conducted to obtain valuable input and modify the design and corresponding functionalities of the CA (Nielsen, 1997; Tremblay et al., 2010; Venable et al., 2016). Therefore, the DPs and their instantiation in the CA were presented, tested, and discussed in two focus groups with potential users (four participants, 59 min. duration) and researchers with different contextual knowledge (software developers, CA/AI experts) (five participants, 91 min. duration) (see Sect. 6). To perform a naturalistic ex-post evaluation, a two-fold strategy is applied to leverage an extensive set of empirical data and gain insights on the efficiency and feasibility of the instantiated DPs (Venable et al., 2016). First, data on characteristics of submitted ideas from real users was gathered. For this purpose, the CA was deployed on a website during a research project involving partners from research and practice in the field of public administration. After initiation of the open call by several project stakeholders, 40 external participants submitted an idea on the topic “Mobility of the Future”. Based on these submissions, the characteristics of the ideas were examined, on the one hand, in semi-structured interviews with four experts from, inter alia, the fields of innovation and product management. On the other hand, a computer-based analysis was conducted to investigate the linguistic attributes of the collected ideas to draw inferences about the affective and cognitive processes of the idea contributors. Second, the CA facilitator and a standard submission form for idea generation were compared to assess the level of engagement and perceived social presence induced in potential idea contributors. Therefore, 221 participants were divided into two conditions to observe one animated mock-up simulating the respective idea generation process. Subsequently, participants completed a questionnaire-based evaluation of the simulation. Step six (communication) will be completed with the publication of this study.

Being part of a multi-cyclic DSR project (see Fig. 1), this research aims to make a two-fold contribution by providing prescriptive design knowledge and a corresponding design entity in the form of an instantiated CA facilitator (Drechsler & Hevner, 2018; Gregor & Hevner, 2013). Besides codifying the functioning and construction of the artifact, the utility character of the generated design knowledge is established via the comprehensive evaluation (Kuechler & Vaishnavi, 2012; Venable, 2006). In the following sections, we elaborate on the delineated steps of the third cycle of the DSR project covered in this publication.

The ex-ante evaluation episode focuses on the formative assessment of the created and instantiated design in the form of an automated conversation-based CA facilitation to purposefully address the identified real-world problem of unsystematized idea submissions due to limited support for contributors during the idea generation process (Sonnenberg & Vom Brocke, 2012; Venable et al., 2016). To evaluate the applicability, operationality, and completeness of the created and instantiated CA design, we utilized EFGs to leverage a rich qualitative data set.

Following the proposed steps by Tremblay et al. (2010), we conducted two EFGs to obtain profound feedback on design-related aspects as well as technical functionalities of the initial CA version. The first focus group (EFG1) lasted 59 min and consisted of four participants, each of whom had participated at least once as a contributor in an open innovation initiative. Accordingly, the two female and two male participants represent potential CA users. The second focus group (EFG2), which comprised five male participants from research with expert knowledge in software development of CAs and/or AI, lasted 91 min. Each focus group was recorded and followed a pre-defined procedure: (1) presentation, (2) demonstration, and (3) discussion. In the first phase, the context and objective of the study was presented. Subsequently, participants individually conducted a click-through and executed functional tests to evaluate the CA during the idea generation task. After that, a prepared guideline with open-ended questions based on the four DPs was utilized to validate and refine the design. Based on transcripts, a qualitative content analysis according to Mayring (2014) was individually conducted by two researchers and resulting codes were continuously harmonized to obtain insights about the DPs and derive opportunities for improvement.

In general, the analysis of qualitative data showed that participants of both focus groups rated the CA facilitator as applicable and operational to record and select ideas in a homogenized format for further processing. The user interaction was described as flawless, intuitive, engaging, and human-like. The participants rated the interactive questioning process as detailed, coherent, and targeted. Furthermore, they reported that the submission of an idea was supported by the transparent progress in the process (e.g., “The process design is designed in such a way that it can be followed very firmly, and it is also developed in such a way that the process can be easily tracked” (EFG2)). With reference to DP1, the participants assessed the requirements to be clearly outlined, the posed questions by the CA to be very goal-oriented, and the process design to be easy to follow. The logical step-by-step approach helped “writing things down, which is good when developing a spontaneous idea” (EFG1). Therefore, participants had “the feeling of being guided toward reaching a goal” (EFG2). Moreover, the participants agreed that “CA’s utterances build upon each other” (EFG2) and are suitable for the task of facilitating the idea generation process. For DP2, participants valued the flexible interaction and possibility to ask questions, although only a few of them used this functionality. The CA’s intention to acquire elaborated input was recognizable, e.g., “when he asked whether I would like to confirm or change specifications” (EFG2). In addition, participants considered the support with optional information about the topic at the beginning of the process to be valuable. The “definitions and further instructions during the process steps were goal-oriented, when asked for” (EFG1). In relation to DP3, the user guidance was judged to be well-managed with clearly formulated, precise, and understandable statements and questions. According to the participants, the content of the messages had a suitable length and language level, keeping mental effort at minimum level. In this context, one participant affirmatively stated that “it was easy to follow, it was very clear what was meant and there was little room for misinterpretation” (EFG2). During the click-through demonstration, the CA reacted correctly with prompt responses which was perceived to sustain the progression through the process. While participants rated the strategy of the CA to counteract silent moments as generally useful, the implementation was rated to require improvement, as one participant experienced a mistakenly triggered reaction by the CA: “I wrote a long text and was already asked before sending it” (EFG2). Regarding DP4, participants stated that they were aware of interacting with a CA. Nevertheless, the text-based and friendly conversation was considered conducive to the atmosphere, as it conveys a sense of humanness. One participant commented: “the natural language equilibrates and pulls it away from a pure technical impression” (EFG1). Furthermore, statements from the CA between process phases were perceived as motivational support. The interactivity of the process was regarded to reduce the initial hesitation of starting to submit an idea. The personal address created sympathy among participants for the CA. In particular, referring to the user by name during the process had a positive influence, as one participant reported: “it gives a personal touch, and I am a person who likes to be called by my first name” (EFG1).

With the tentative CA version, the completeness of relevant design aspects could be demonstrated. In addition to confirming insights, focus group members highlighted potential for improvement related to support behavior and technical features of the CA. To increase the advantage of support upon request, the CA should clearly indicate how and for what purpose. The silent moment should not be triggered too early when users are actively making entries, as this unnecessarily interrupts the writing process. From a technical perspective, “the interaction capability is expandable” (EFG2). Accordingly, the language model requires fine-tuning, since intents were sometimes recognized incorrectly and participants were occasionally offered process termination. To encourage users to write extensive and detailed ideas, the entry field should be larger, since “it is better if one has the possibility to see the multiline text” (EFG2).

For the ex-post evaluation, an adapted and improved CA facilitator was implemented based on the findings from the demonstration. To gain insights into the efficiency and feasibility of the instantiated DPs, we conducted a naturalistic evaluation of the final artifact (Creswell et al., 2003; Sonnenberg & Vom Brocke, 2012; Venable et al., 2016). To this end, we completed two field studies and applied various evaluation activities. On the one hand, we deployed the CA facilitator on a website to gather submissions from real users and subsequently analyze the characteristics of the ideas (see Sect. 7.1.). On the other hand, we assessed the feasibility of the proposed solution to engage contributors and provoke a perception of social presence by comparing the levels of engagement and social presence of CA facilitation with a standard submission form.

Organizations strive to leverage external knowledge and expertise by applying open innovation approaches to promote their innovation capability. To manage idea platforms for the outside-in process in such a way that prospective contributors are motivated and supported to voluntarily submit an idea and the large number of emerging ideas can be efficiently selected, we propose a design for a CA facilitated idea generation process. Building on the vast body of theoretical knowledge regarding the concept of facilitation, we derive design knowledge to determine purposeful characteristics and behavior of a CA facilitator. By evaluating the instantiated design knowledge in a dialog-based CA facilitator for idea generation, we provide results regarding the nature of ideas and characteristics of the process. The evaluation with knowledgeable experts and a computerized linguistic analysis revealed homogeneous idea contributions with a constant level of detail, a satisfactory level of comprehensibility, and a high analytical as well as logical character comprising outward-looking words. Furthermore, the questionnaire-based evaluation of idea generation process showed that CA facilitation induces a higher perceived engagement and social presence among contributors during the idea generation process compared to conventional form-based interfaces. In the following, we elaborate on the multifaceted implications of these findings.

First, the presented design demonstrates the integration of the facilitation concept into state-of-the-art CA technology. For this purpose, following Bostrom et al. (1993), we considered all facilitative acts to leverage support for idea contributors during task processing (task), for accomplishing associated goals (process), and to conduct a socio-emotional interaction. Thus, we integrate and extend previous approaches to CA facilitation, as these have so for examined different aspects in isolation such as supportive behavior for idea generation (Wang et al., 2007) and proactive prompting of desired behavior, e.g., the elaboration and reformulation of input (Louvet et al., 2017; Tegos et al., 2014, 2015). As the results of the evaluation of contributors’ textual language style suggest that CA facilitation is related to a fact-based enrichment of information and that idea contributors are emotionally engaged and apply problem-solving and analytical thinking, we imply that CA facilitation may have a positive influence on idea generation in the context of open innovation. As a supplementary point, it should be noted that, in contrast to emotionality, the dimension of analytical thinking was consistent across all facilitative process steps. Consequently, linguistic analysis of the text data denotes that the idea contributors used analytical writing for idea text, albeit using a positive language style that suggests they were emotionally engaged. Under the given conditions, this can be associated with the CA’s goal- and productivity-oriented behavioral capabilities. Furthermore, the data lends support to the finding that idea contributors were more focused on others than on themselves during the interaction when generating the idea text (Moore et al., 2021). This is a promising finding as it may indicate that humans focus on their interlocutor in this context, even when the partner is deliberately artificial but uses human language patterns.

Second, our results show that the idea generation process can be designed in such a way that idea contributors are more engaged compared to conventional form-based interfaces. This is a valuable insight in the context of open innovation processes (e.g., crowdsourcing, idea contests), as organizations struggle to motivate voluntary and unpaid idea contributors to start and complete submissions (Bretschneider, 2012). Besides manipulating the presentation of the task and goal (Benz et al., 2019; T. de Vreede et al., 2013), the implementation of a task-focused CA that facilitates the idea generation process in a human-like fashion constitutes an additional effective method to increase users’ engagement. This insight is supported by the perception of the focus group participants who stated that CA facilitation reduced their initial hesitation to initiate the idea generation process and provided goal-oriented guidance in the process. In addition, the statistical analysis based on the survey following the simulated idea generation process has shown that significant differences exist between the CA facilitated and non-facilitated idea generation process in terms of engagement and perceived social presence. The fact that engagement and social presence correlate is not unexpected, as the concepts are closely linked. Interestingly, however, differences were observed between the two conditions with respect to significance in the correlations of the two constructs with the variable gender, which could be relevant for the further development of an individualized CA behavior.

Third, the generated design knowledge and design entity in the form of the CA facilitator provide a novel approach to enhance the efficiency of idea selection through an improved idea generation process. More specifically, supporting idea contributors reduces the likelihood of a heterogeneous pool of ideas with low levels of detail, elaboration, and likelihood of implementation. This idea generation approach provides organizations with the option to flexibly implement adjustments in the process and CA’s facilitative acts to tailor the support of contributors to a specific task and determine a pre-defined set of required information. Accordingly, the facilitation of the idea generation process can serve as a preparatory step for a systematic idea selection process. Thereby, our findings address relevant questions about organizations’ efficient management of large numbers of collected ideas with restricted resources in the context of open innovation initiatives (Blohm et al., 2013; Merz, 2018).

As part of a multi-cycle DSR research project, this study presents a solution to elevate organizational idea generation processes on idea platforms with AI-based CA technology. While idea generation facilitation is critical to innovation, organizations struggle to leverage this potential on idea platforms. So far, large amounts of ambiguous, imprecise, and incomplete ideas hamper organizations in selecting ideas with potential for further processing. To address these challenges, we built on the facilitation concept to iteratively design and instantiate a scalable CA that facilitates individuals during their idea generation. Evaluation results suggest that the natural, dialog-based interaction encourages and engages idea contributors to provide clear, detailed, and complete ideas, which deliver a suitable grounding for the essential follow-up selection of textual ideas in organizations.