Bots in software engineering: a systematic mapping study

Managing software repositories

Software repositories have heavily contributed towards collaboration in software development. Repository platforms provide utilities for code documentation, discussion of issues, running continuous integration (CI) scripts and executing version control commands. Due to the standardized workflow processes of pull-based models, usage of bots is facilitated in such environments to automate recurring tasks. Bots have been increasingly adapted for managing software repositories since the year 2013. In a sample of 351 popular GitHub projects it was found that at least $\frac{1}{3}$ of them make use of software bots (Wessel et al., 2018).

The general workflows of collaboration models in today’s most common versioning system (Git) is based on pull requests issued by the contributors. Every pull request contains one or more commits to the specified branch of the repository. These commits could be a fix for an issue, a feature or documentation. Some repositories are also configured with continuous integration (CI) pipelines to verify whether the received pull request passes various levels of tests so that the pull request can be merged with the specified branch. We would not consider a CI tool which merely checks for build failures to be a bot, but there exist bots which are activated when there is a build failure. Bots used in software repositories are classified by Wessel et al. (2018) into 12 categories based on the tasks they perform. These are license agreement signing, CI failure reporting, automated code review (code smell checking, test coverage, style checking), welcoming newcomers, assigning code reviewers, merging pull requests, fixing vulnerabilities, automated testing, automated software builds, controlling dependencies, issues creating, and benchmarking. Apart from these, bots can also be used for documentation, code cleanup, and code analysis. To better understand the usage of bots in software repositories, Dey, Vasilescu & Mockus (2020) studied the types of commits made by bots in GitHub. The results show that bots are used for editing documentation and configuration files much more than source code files. Specific tasks performed by software repository bots are discussed in more detail in the following.

Q&A Bots

There is a great need to provide improved techniques for information retrieval and exploration in software engineering (Xu et al., 2017). Bots can also serve as interfaces to existing knowledge bases, aiding the developers to access them effortlessly.

Software development environment

Bots deployed in developers’ local environment function as assistants supporting them in their specific tasks from modeling to testing to requirements engineering. In contrast to software repository bots that are more concerned with the collaborative nature of the software project, these bots are like personal assistants that are run on development environment and serve the developer.

Collaborative bots

Collaborative bots are conversational chatbots which run on communication/chat platforms such as Telegram, Slack, Twitter, or Discord. As the name suggests, these bots foster collaboration among the developers. Apart from the bots which converse with a single user, these bots are aimed to be a part of a discussion among a community. Being reactive, they retrieve useful information during the discussion, whereas proactive bots provide interesting tips based on the topic of discussion. As most of the collaborative bots are reactive, they do not interfere in the discussion among the peers.

SOCIO (PerezSoler et al., 2017) is a conversational modelling bot used for collaborative modelling among various levels of developers. SOCIO helps to create and design software models through Natural language commands. It is deployed on Twitter and Telegram channels. This facilitates bot developers and designers to model collaboratively through social media channels. They improved the bot further (PerezSoler, Guerra & de Lara, 2018) by considering the decision-making processes for multiple possibilities to reach a soft consensus among everyone. In the study by Ren et al. (2020), SOCIO has been evaluated against an online GUI-based tool and SOCIO performed great in terms of efficiency and satisfaction, and equivalent in effectiveness and quality. In contrast to traditional modelling, PerezSoler, Guerra & De Lara (2017) and PérezSoler et al. (2019) proposed social conversation with model development through chatbots and conversation as a platform (CAAP). Feasibility of the work is demonstrated with a case study that focused on collaboration in natural language, multi-platform, mobility, and traceability.

Taskbot (Toxtli, Monroy Hernández & Cranshaw, 2018) is a communication bot deployed in Microsoft Teams, where users can tag the bot and create tasks which will be communicated with other users and keeps remainder. With an overall positive score, some negative feedback included are unable to recognize people names without proper syntax, highly frequent reminders, handling multiple threaded conversations, and human ambiguity. Cerezo et al. (2019) developed a bot which recommends experts in a software community in a Discord chat. The results unfortunately turned out negative, as the bot did not consider the social aspect of carrying out a conversation. As no help instructions were provided, the bot was found non-user-friendly. Bots also function as an interface to access software visualizations. Bieliauskas & Schreiber (2017) developed Sofia and deployed it in a communication platform for developers to interact with visualizations. The developer can directly ask Sofia about a ticket, and the bot actively serves a customized visualization based upon the request. Being on a communication platform, Sofia also monitors passively when two or more developers are discussing a subject and provides a suitable visualization for them.

A significant feature in collaborative bots is that they can passively listen to the conversations and perform actions according to their functionality. GitterAns (Romero, Parra & Haiduc, 2020) is a bot deployed on Gitter (https://gitter.io) channel that provides solutions to the problems faced by developers. The bot passively listens to their conversations and when troubleshooting problems are discussed in the chat platform, GitterAns identifies which turn of the conversation is related to the problem and provides a solution appropriately. To enhance the productivity in a collaborative working environment (CWE), Corti et al. (2019) adopted a virtual assistant to assist developers by leveraging Internet of Things (IOT) and instant chats. The bot listens to the chat messages of the developers, interprets them and executes the respective actions. The core aspects of the assistant are collaboration, communication, information management and content over structure. Most collaborative bots are reactive, but being reactive in a communication channel might not mean that the bot has to respond to the main line of discussion. There are also bots, which attend to specific keywords in the overall topic of discussion. Sannikova (2018) deployed a chatbot on Slack, and it is connected to a restaurant menu page. Even though the main topic of discussion will be software development, the bot only responds to the topic of food. The bot returns the day’s menu, When the developers query about food. Being a reactive bot, it will not interfere with other discussion topics.

During collaboration on closely related tasks, it is often hard not to have conflicts with the code developed by others in the team. Code conflicts are usually managed with best practices, but using bots for eyeballing the conflicts eases up the work of developers. Sayme (Paikari et al., 2019) takes care of code conflicts by notifying the developers proactively and reactively. It constantly looks for conflicting files and notifies developers on such conflicts. One can also ask Sayme about the status, and it provides results reactively. The most notable thing about Sayme is that it detects code conflict even when the conflicts are in different files but shares common code execution like function call and definition.

In summary, collaborative bots combine conversations and commands, thereby making bots inclusive of the development community. They are sensitive to their social aspects as they are established among a community of developers. Bots with reactive characteristics only function when the developer demands it. This serves the benefit of avoiding unnecessary notifications and noise.

Chatbots

Bots involving natural language conversation falls under Chatbots, previously called Chatterbots. Chatbots are recently being employed in various domains from restaurants to e-commerce to customer helpdesks. In this section, we discuss chatbots that are being used in the software engineering domain. Based upon how they converse, chatbots are either rule-based or generative. As the name signifies, rule-based are formed of rules or intents. The number of intents is fixed, and the user’s input could be only classified into one of these intents. On the contrary, generative models do not rely on fixed set of intents but are learned via machine learning or deep learning (Sutskever, Vinyals & Le, 2014; Cho et al., 2014). And the conversation responses are generated based on the model weights and input sentences. Generative models are still in research phase, as the responses are unpredictable, whereas rule-based models even with limited functionalities are found in most of the current products due to its reliability.

Rule-based or Retrieval-based chatbots consists of three main components: Natural language understanding (NLU), Dialogue management, Natural language generation (NLG). NLU may sound sophisticated, but essentially NLU classifies the input to one of the many listed intents and extracts entities. Dialogue management takes care of the path of the conversation. NLG generates natural language responses through templates. When the chatbots are expected to just chit-chat, no knowledge bases are needed as the commonly used chit-chat combinations are written in the Dialogue management itself. Some popularly used NLU components are Google Dialogflow (https://cloud.google.com/dialogflow), Microsoft LUIS (https://www.luis.ai/), IBM Watson (https://www.ibm.com/watson), Rasa (https://rasa.com), Xatkit (https://xatkit.com/). These components provide both NLU and dialogue management services. Abdellatif et al. (2021) evaluated the NLU components that are suitable for software engineering tasks. The comparison resulted in IBM Watson being ranked as the best for intent classification and entity extraction, whereas Rasa ranked the best for confidence scores.

To study about the activity of Chatbot topics in SO community, Abdellatif et al. (2020) mined SO for questions and answers related to chatbot tags to answer three questions, topics in the area, types of questions and difficult questions. The common topics were integration, development, NLU, user interaction, and user input. The difficult to answer questions were related to training the models. Training NLU model involves suitable datasets, frameworks, type of model. In the view of developing virtual agents (VA) in SE, the NLUs have to be trained with intents and examples from software engineering field. Since the SE-dataset is domain-based and scarce, the work by Wood, Eberhart & McMillan (2020) focused towards applying transfer learning on a global encoder –local encoder model to the task of classifying dialogue act types. The global encoder is trained on a generic dataset such as AMI business corpus and the local encoder is trained on a manually annotated small software engineering dataset.

To develop bots in SE, it is essential to design the bots to suit the needs of the developers. Wizard of Oz (WOZ) experiments are conducted to understand the expectations of the developers and design bots accordingly. One such WOZ experiment is conducted by Wood et al. (2018) to study about different speech act types used by software developers on bug repair. A supervised model is trained to classify them. Among all the speech act types, developers asked clarification questions the most. Another WOZ experiment by Melo et al. (2020) studied how chatbots will aid software developers in day to day tasks. The results demonstrate that the use of chatbots will be certainly helpful in software engineering and the role of chatbot as a guide is preferred over executing tasks for the developers. Participants also liked the contextual information that the bots possess on what the developer is currently working on.

Chatbots perform as conversational interfaces to existing databases, thereby allowing the user to access the information handily. The bots are linked to such databases and NLUs are designed accordingly. Matera & Castaldo (2019), used chatbots for data exploration, where the data is stored in a relational database. The bot replaces the SQL command line with a conversational interface. By the use of conversational relationships, qualifiers and values, the bot functions with rule-based system but with more relaxations and flexibility to the database structure. Chatbots engage developers by being an interactive interface to static documentation pages. OpenAPIBot (Ed-Douibi, Daniel & Cabot, 2020) is a chatbot that helps to provide information about the desired API. The bot offers interactive capabilities to the API navigation in contrast to the simple descriptive API documentation. The bot helps developers to understand REST APIs by asking questions using natural language. The implementation uses XATKIT and powered by Dialogflow for the NLU part. In software crowdsourcing tasks, bots apply information retrieval to help users find appropriate developers for the task. CrowDevBot (Ni et al., 2019) is a conversational bot used for Crowdsourcing. The bot assists users in completing crowdsourcing tasks via conversation, rather than tedious mouse clicks. To obtain domain information, the chatbot is linked to a database called Software service knowledge base (SSKB), which contains information about projects, required skills and experts.

Chatbots are applied in educational platforms to teach students certain skills. Boubekeur & Mussbacher (2020) designed an interactive domain modeling assistant that assists students with Domain modeling through gamification and live feedback. The functionality of the bot is described with three modeling scenarios, mistakes made with software engineering terms, placing attributes in wrong class and mistakes in player-role patterns. Chatbots are used to facilitate non-technical users as well to use model-driven engineering. iContractBot (Qasse, Mishra & Hamdaqa, 2021) is a chatbot developed with XATKIT framework for developing smart contracts. Based on Model driven development, the bot converses with the developer and develops model from Natural language interactions. PérezSoler et al. (2020) synthesized a chatbot using xatkit framework for model query. With the help of domain-meta model and natural language configuration, the xatkit generates an executable chatbot, which is then used to query models. The actions are written for every intent as execution rules.

Chatbots that help developers are not restricted to code related tasks, they also take part in productivity-based works. Amber (Kimani et al., 2019) is a proactive chatbot, that takes care of task scheduling, task reminders, managing time spent outside of work in social media platforms. It helps to review the scheduled tasks by the end of the day. It uses Microsoft bot framework and conversational learner framework. Though the bot received average ratings, the participants made changes to their routine to be more productive and healthy.

The fact that chatbots being used as an interface for accessing database in many works show that natural language interactions are actively researched over conventional methods in information retrieval. The availability of various NLU and dialogue management frameworks surely contributed to the studies in this topic. Even with the restricted functionalities of retrieval-based NLU techniques, the results are positive overall. Thus, improvement in NLP representations and ML models with resulting advancements in NLU will push the boundaries of chatbots further.

Voice assisted bots

Voice assisted bots are activated using natural language voice commands from the developer. Automatic speech recognition systems (ASR) are integrated with the existing chatbot frameworks to extrapolate the usages to spoken language capabilities. As bots are aimed to work along with the developers in a bot-ecosystem, these bots accelerate the development in such directions by leveraging voice capabilities.

Devy (Bradley, Fritz & Holmes, 2018) is a conversational developer assistant, which executes version control commands. It works as a replacement to the traditional command line interfaces. Devy is built using Amazon Alexa framework, and it is deployed in Amazon Echo device. Robin (Da Silva et al., 2020) is also a voice controlled bot for developers to access repository-based information. Compared to Devy, Robin focuses on questions useful for developers during collaboration. Robin has been implemented with Alexa and Google Assistant. Smartadvisor (Sharma et al., 2019a) is a conversational interface(supports both text and speech inputs) to aid developers with information. By connecting with a data exhaust like IDE, version control and CI, the bot could provide information to the developer via a guided conversation.

Speech interfaces are not just used for a direct transcription to the existing commands, rather with the power of NLU, they could be directed to initiate actions. This opens up many opportunities in Human–computer Interaction (HCI). IslandViz (Misiak et al., 2018; Seipel et al., 2019) lets developers visualize complex software architectures in 3D via Augmented Reality (AR). Speech-based commands are used to navigate and explore specific parts of the visualization in Microsoft Hololens. With Speech and gesture control inputs from the user and the corresponding visualization changes in the View, it provides an engaging interface to the users for exploration. Rasa is used as the chatbot framework and Microsoft in-built ASR is used for Speech-to-text (STT). Visualization of software analytics using city metaphor in AR is also done by Baum et al. (2020). The users interact with the visualization via gesture and voice controls. Although the voice control worked well from a technical point of view, it was rejected by the users and the virtual on-screen keyboard was also unsuitable.

Apart from exploring code files, code is also written using utterances. Convo (Van Brummelen et al., 2020) is a voice-based conversational programming platform. Developers issue natural language commands via speech utterances, and they are converted to respective code. Convo contains voice-user-interface (VUI), NLU module, Dialogue Manager and a Program editor (PE). The number one recommendation of the work was to improve the speech recognition system, as the accuracy of the ASR directly affects the conversational programming. Voice-Driven Modeling (VDM) has been envisioned by Black, Rapos & Stephan (2019) to adapt voice-driven programming to Software modeling. The VDM framework consists of ASR, NLP and context-specific modeling blocks. Being a position paper, the framework is aimed to be useful for software modelers to improve their modeling efficiency, to domain-experts with limited modeling knowledge, and to modelers with disabilities.

Comparatively, few works have been done on bots with voice assistance. These bots depend upon the performance of ASR systems, and speech recognition is a huge research topic of its own. These recent works on such a multidisciplinary topic will open up new research ideas and possibilities towards the use of voice activated conversational AI in software engineering.

HCI concepts

To function optimally, designing a bot is as significant as developing the bot. As they function as a medium between developers and software programs, the design concepts used in human–computer interaction (HCI) takes prominence. Imparting suitable design principles for the bot and in the environment helps the bot to achieve efficiency for the intended task.

The socio-technical aspects of bots are addressed in Lebeuf, Storey & Zagalsky (2017). The work lists all the social frictions that might occur in Teams interactions, Teams interaction with Technology and individual interaction with technology. The work also listed all the existing bots which tried to reduce these frictions. Pinheiro et al. (2019) surveyed developers on the motivation and challenges using bots. Around 65% of the participants worked alone while developing the bots. The survey indicates that, most of the developers created bots for their personal needs and to assist them in the workplace. There were considerable difficulties in bot implementation and finding tools for bot development as well. Washizaki (2020) presented a vision called value co-creation of Software by AI and developers. When applying AI to development process, aspects such as automation, accuracy, robustness and explainability have to be considered. Cooperative activities and interactions are necessary among developers, among AI, and among AI and developers towards value co-creation.

Digital nudging is a concept from behavioral science, where nudges and notifications guide developers towards best practices. Brown (2019a) and Brown (2019b) applied these digital nudges in tool recommendation to suggest static code analysis tools for Java projects. By integrating Nudge theory in a bot, Brown and Parnin developed nudge-bot (Brown & Parnin, 2019), which provides recommendations about better software practices to university students via digital nudges in both active and passive fashion. Further work (Brown & Parnin, 2020) focused on developer recommendation choice architecture of the bot such as actionability, feedback, and locality of the nudges. Results from the experiment show that, developers prefer actionable recommendations to non-actionable pull request comments.

Wessel (2020) proposed the concept of a meta-bot which manages all other GitHub bots. The author suggests a study to understand how developers view bots, and provides guidelines to create the metabot. Using Design fiction as the concept, the study has been conducted among software developers to see if the vision of the meta-bot suits the need of the developers. Wessel et al. (2020b) carried out another study among OSS maintainers to check the comfortableness while using code review bots. Results show that the main reason to use such bots are better feedback to developers and reducing maintainer’s effort. Bots reduce the manual labor from maintainers to maintain a project. They also report on improvement in code quality, increase in contributor’s confidence, decreased time to close a pull request. Liu, Smith & Veeramachaneni (2020) studied the behavior of Bots in OSS projects and how it affects the interaction between bots and humans. Design principles are proposed for a trouble-free human-bot interaction. They are, to be robust and stable, ensure transparency, provide simple responses, reduced interruptions, rich user interface, personalized behavior and not to overuse bots. These principles should be used as guidelines to evaluate the user experience of a bot’s interaction behavior. Consolidating the design of repair bots, Van Tonder & Le Goues (2019) proposed 6 principles on how repair bots should be built. The principles are based on syntax generality, semantic validation, testings and static code analysis. The work also pointed out the importance of integration of human workflows in the automated system.

For a generic NLU model on a retrieval-based chatbot, bots execute the same intent every time when different expressions of the same query is posted. Users get annoyed of this behavior, as bots are insensitive to the expertise of the users. Performobot (Beck et al., 2020; Okanović et al., 2020), which runs load testing in Software systems tackled this issue by having intents at various levels. The bot lets expert, novice and intermediate users to execute the same action via different intents. This design feature gives the flexibility to access the bot according to the level of expertise.

Wessel et al. (2021) did a study on bot developers and maintainers to check the disruption of the bots to the community. Noise is found as the recurrent and central challenge, which affects both human communication and development workflow. Noise is viewed as a bot behavior, such as the number of bots in a project (overpopulation), information overload of the bots –which interferes with the collaboration model of humans on social coding platforms. When contributions made by bots are without proper explanations, transparency is an issue on why such fix has been made by the bot (Serban et al., 2021).

When developing bots for collaboration, it is important to consider the communication style of the developers. Taskbot (Toxtli, MonroyHernández & Cranshaw, 2018) had issues with identifying people names without proper syntaxes, human ambiguity and handling multiple threaded conversations with their corresponding tasks. SOCIO (PerezSoler, Guerra & de Lara, 2018) is used in a collaborative environment for modelling, where achieving a unified consensual decision among different expert groups is hard. SOCIO ensures reaching a soft consensus between modelling experts and domain experts on the task of domain-specific language modelling. Saadat, Colmenares & Sukthankar (2021) developed a bot classifier to identify bots in GitHub. They then conducted a study among humans-bot teams and humans only teams to examine the nature of workflow. The human-bots team had an active communication via issue comments throughout the development process, compared to the humans only teams. “Bots force human members of the team to discuss issues between different stages of the workflow”

HCI concepts from cognitive science such as behavioral sciences are used in the design of bots. Studies on behavior of bots provide insights into how they are perceived by the developers and maintainers. Based upon the application, appropriate design features have to be considered, as discussed in the collaborative environment section. Noise (too many notifications and untimely interruptions) and Transparency (explainability of the actions) are recognized as notable issues.

Bot persona

Bots emerged as a way to automate scripts, but with successful integration of Natural language tools they are now used to execute tasks on the behalf of humans via commands and conversations. For such interactions, it is essential for the bots to possess a persona. Persona is the characteristics given to the bot for a smoother social interaction. This comforts and even excites the humans to have a conversation with a bot. These personas are designed according to the task at hand so that it emulates the behavior of a person in that place. The bot’s persona has to be stable and consistent throughout the entire conversation.

Bots are expected to emulate social roles like humans, when bots are to be integrated into communities. This motivated Seering et al. (2019) to study about multiparty bots. Various bot personas such as antagonist, archivist, authority figure, dependent, clown, social organizer, storyteller are discussed. Kuttal et al. (2020) designed a conversational agent (CA) for pair-programming task through WOZ study. Results convey that the CAs are to be developed with social intelligence, to include embodiments (face and voice) and trustable traits. Moreover, when Agents interrupt humans, humans don’t like it unless there is a high-error probability in their computation. Developers have a welcoming and forgiving attitude towards software bots when they have a positive name and character (Monperrus et al., 2019).

Designing the persona did not purely rely on the sociological aspect of the bot. When the personas are linked with the capabilities of the bot, the persona could also reflect the skill of the bot. Erlenhov, Neto & Leitner (2020) classified Devbots based on the personas as chatbot persona (charlie), autonomous bot (alex), smart persona (sam). Currently, lots of bots with charlie and alex personas are present but very few sam type bots. All classes of bots are in some way prone to producing noise. Paikari & Van der Hoek (2018) compared the chatbots for developers based on type, guidance, direction, interaction style and communication channel. The Task-oriented chatbots don’t engage in random chatter to build relationships, as they are mainly built for executing specified tasks. None of the chatbots with behavior evolution are designed specifically to support software development. Even in collaboration, chatbots did not exhibit an engaged interaction style, but just informational and automation chatbots existed. But depending upon personal preference, the user expectations may vary. In the work done by Matera & Castaldo (2019), even the chatbot which is built for data exploration received feedback to have chit-chat nature for a better persona.

Bots sometime take the role of a developer and try to act as one among the other human developers. This leads to the discussions in the ethical point of view. There are valid arguments on both sides whether bots should disclose the identity or should it be disguised as a developer. Being in the disguise, bots have the trust of the fellow workers and are considered as one among them. This might raise questions on accountability. As being explicitly disclosed as bots, they are not taken seriously by the developers due to the limitations of the bots. Murgia et al. (2016) conducted a study by letting a Q&A bot run in the SO site for 90 days. In the first phase it was disguised as a human, and it performed well. But in the second phase where it was explicitly mentioned as a bot, the bot received a lot of downvotes, poor comments and got banned in 25 days. From the qualitative study by Wyrich et al. (2020) on a refactoring bot, the conclusion was that bots are accepted as bots, even though they are considered critical compared to human developers. This concurred with the Dagstuhl seminar discussion (Storey et al., 2020) “The bot should not hide its identity from the users”.

Apart from the ethical viewpoints of the developers, how bots should be disguised, it is vital to the user community to recognize whether the user is a bot or not. This ensures better transparency in the bot-ecosystem. Based upon the actions executed by the identity, it could be identified as bot or not. To identify bots, Dey et al. (2020) proposed BIMAN (Bot identification by commit message, commit association and author’s name). Based upon the characteristics of the commits, they are scanned for unusual behaviour which are abnormal for human developers. They also considered the time and spike of the bot’s activity for better identification. Golzadeh et al. (2020) did a similar work on identifying bots from human contributors based on the comment patterns in pull requests. Based on a combination of Levenshtein distance and Jaccard similarity of the comments, they are clustered and bots are identified. It is based on the hypothesis that bots repeatedly use a certain set of words. Further work on the bot identification by Golzadeh et al. (2021a) used encoded representations with ML methods for classification. Based on the structure of comments from pull request and issues the bots are identified by training an ML model of vectorized words with Bag-of-words (BOW) and Term frequency Inverse document frequency (tf-idf) techniques. Golzadeh et al. also published a ground-truth dataset (Golzadeh et al., 2021b) of 5k GitHub accounts with bots tag to promote research and evaluation of bot identification methods.

In summary, the persona of the bot has to be designed in accordance with the role and functionality of the bot. Oftentimes, it is also restricted with the capabilities that the bot possess. Social aspects of conversation are expected even from the task-oriented bots. Should bots disclose their identity or is it ethical to be in disguise of a human, still remains as an interesting discussion. Nevertheless, the tools, which are used to identify bots are actively developed.