A review of natural language processing in contact centre automation

CCs traditionally known as call centres are among the most important contributing factors to customer relationship management (CRM) and serve as the primary interface between organizations and their customers. Today CCs are referred to as worksites where CSRs interact with customers over omnichannel platform that integrates channels such as phone, email, fax, letter, website, live chat, and social media [8]. Typically, customers use three different communication devices to communicate with an organization’s CC: traditional phone, computer (laptop or desktop), and smartphones. In the early call centres, the only communication channel was a voice but now because people are more tech-savvy and interactions are dominantly widespread in the personal communication/social media market, CC has become omnichannel. Omni-channel CC is a progression from the multi-channel model where various channels of communication are supported and integrated, such as voice chat, video chat, emails, SMS, webchat, and social media messaging [10]. Voice (phone) channel is the most used communication channel in a CC and is usually categorized into two operational modes: inbound and outbound. Inbound is when customers call into CC and outbound is when CSRs call customers. Figure 2 provides an example of various channels offered by a modern contact centre today.

The two main goals of CCs are improving customer satisfaction and reducing operating costs, essentially providing efficient service at a reasonable cost. There are trade-offs in achieving these two goals concurrently as they are perceived as incompatible with each other [11]. It is estimated that 70% of all company interactions are from the CCs [12]. Another report highlights that it costs organizations\1.3 trillion every year on 265 billion customer service calls globally [13]. Thus, automating even a fraction of the interactions handled by the CSRs can generate tremendous cost savings. Organizations have reduced operational costs by focusing mainly on automating critical processes such as automatic call distribution using touch-tone interactive voice responses (IVRs) or outsourcing CCs to other countries with lower labour costs, which accounts for 60–80% of total CC expenditure [14]. However, this has jeopardized customer satisfaction and only resulted in high customer churning and employee attrition rates. Outsourcing has several issues including language and culture, time zone, geographical and legal, and political instability [15]. CCs historically have aimed of achieving the lowest cost of customer service delivery. This is why most businesses relocated their CCs to countries that are inexpensive concerning operations costs. CSRs are not valued which leads to high agent churn, ultimately adding cost for organizations. Key performance indicators (KPs) focused solely on metrics related to cost. Businesses are after the shortest possible average handling time (AHT) and customers are treated less as individuals by subjecting them to generic “scripts” and keeping them on hold for longer periods. Repeat calls have become common as CSR’s focus on minimizing time rather than fixing the issue, meaning issues are often not resolved. Further, customers shift to competitors providing better services, and replacing lost customers with new ones becomes far more expensive than retaining the existing ones.

In addition, touch-tone IVRs have led to problems like complicated menus, homogeneous service, and poor design of user interfaces, and most importantly, customers feel neglected [16]. A survey conducted reports that customers felt frustrated and angry due to the widespread adoption of IVR systems by the CC [17]. As a result, customers seek CSR assistance at the first opportunity, thereby increasing call-waiting time. While touch-tone IVRs are widespread, speech-enabled IVRs have made substantial headway at replacing them. A study reports that customers prefer natural language-based call routing over usual touch-tone cumbersome menus, therefore delivering significant cost savings and meeting customer expectations [18]. The study also shows that about 20% of the callers who opted for touch-tone-based IVR system routing do not get routed correctly to the service department, ensuing in transferred calls subsequently.

In the paper from [19], it is emphasized that skill-based routing is an important but understudied research area. Wrong routing decisions lead to customers being transferred to the wrong department which is a major concern for both customers and businesses. The study from [20] shows that in an outbound call centre context, their proposed algorithm for call scheduling improves the Right Party Call (RPC) rate by 10–15%, which could mean huge savings on cost for a large CC. A study from Bain & Company reports that for most organizations, a 5% increase in retaining customers could mean a 25% to 95% increase in profit [21]. However, unlike costs or productivity, it is difficult to measure customer satisfaction. Most CCs conduct a manual survey with a small group of customers, typically via a telephone interview or mail-in form. As manual surveys are costly to be conducted on all customers, only 1–5% of customers end up being surveyed weeks after their interactions [22]. A study has also found that for decades response rates have been falling across all types of survey research [23]. Hence, conclusions drawn from manual surveys are not very reliable and do not reflect the correct picture of overall customer satisfaction.

Organizations have now realized that by focusing too much on minimizing the direct cost of running CCs, they failed to factor in the opportunity costs. Thus, resulting in frustrated customers, falling customer loyalty, loss of valuable cross-sell and up-sell opportunities, and the squandering of customer feedback by treating CCs as an afterthought or as a silo that is measured outside the range of corporate goals. It has become paramount to roll out efficient ways by which the expectations of customers and CSRs are realized. It is not sufficient to just have a skilled-based group of agents in the CC; the total customer experience at every point of contact has to be addressed to create a sustainable experience [24]. Therefore, organizations that recognize the changing customer needs and market have already begun the process of applying advanced NLP techniques into their CCs. Not only it can provide a strong and engaging customer experience and a better understanding of their intent but it also offers cost-effective ways to add value to current customer service offerings, decrease churning rates, and increase sales.

The COVID-19 pandemic has accelerated many trends that were due to happen soon. Remote working agents, digital or social media self-service, messenger bots, and ML have started to replace previous business processes. For end customers, it means a well-crafted service, boosting their experience to a level closer to their expectations. Customers can move swiftly between channels and pick up any error-free, frictionless channels. New technologies such as chatbots are rapidly becoming the norm—which orchestrate interactions in an automated way without human intervention. It is no longer about effectively managing telephone contacts at a lower cost but more about delivering end-to-end experience, using advanced technology to stimulate advocacy and loyalty. However, there is a range of challenges that can slow that acceleration.

Many scholars have recognized the lack of data integrity [25], lack of conjoint between CRM and CC data [26], and complexity of CC’s back-end operations [27] as the main challenges of CC. Another issue is the work and effort required to program on the back end that is not fine-tuned and well-structured [27]. As a result, the majority of the data remains in an unstructured format, thus reinforcing the significance of adopting modern techniques that can efficiently analyse unstructured data. One possible way of addressing this is using Big Data tools and technologies and the work from [28] is a good example where they propose an automated system to measure call centre performance. However, the main challenge mentioned by them was the lack of call record corpus. Although existing literature holds practical methods and examples for mining semi-structured and unstructured datasets, the issues of unclean data and heterogeneity within the CC domain remain unaddressed and a paucity of studies remains prevalent. In addition, enhanced NLP applications have progressed significantly and taken the market by storm but there are still challenges that need to be addressed [29]. Organizations need to address these limitations and put in place processes that bridge the gap towards CC automation.

The history of NLP goes back to the late 1940 s when the term was not even in existence; however, work on machine translation had started. Weaver and Booth started one of the earliest Machine Translation projects in 1946 based on expertise in breaking enemy codes in World War II [33]. It was their idea of using cryptography and information theory for language translation that inspired many projects. It was not until the early 1980 s computational grammar theory became a prominent research field, which concentrated on understanding logic, meaning, and extracting beliefs and intentions [34]. By the end of the 1990 s, powerful all-purpose sentence processors such as SRI’s Core Language Engine [35] and Discourse Representation Theory [36] came into existence, offering practical resources, grammars, tools, and parsers for analysing natural language. The use of statistics became a major theme in the 90 s, involving automatic summarization and information extraction and efforts from cross-disciplines became necessary to properly address the challenges of NLP [37, 38]. Until 1990, the progress was slow due to computational and power limitations and research work was mainly in the development of NLP concepts and machine translation. Subsequently, other NLP application areas started emerging and are now significantly researched such as speech recognition [39]. Recent NLP research has evolved majorly with the use of advanced ML algorithms gaining a lot of prominence, especially complex deep learning techniques [40‐42]. Current NLP work is dominated by recently proposed NLP models by Google, OpenAI, Toyota, Facebook, and Carnegie Mellon University such as TransformerXL, GPT versions, BERT, XLNet, ALBERT, RoBERTa, and Wav2vec 2.0. They have proven superlative when compared with traditional models. This has also opened many new opportunities for businesses and the open-source community. The reason for their success is due to their fast processing speed and completeness in representing the language.

NLP helps in organizing natural language and solving a wide range of problems—Machine Translation, Text Summarization, Named-Entity Recognition (NER), Topic Modelling and Topic Segmentation, Sentiment Analysis, Speech Extraction, Semantic Parsing, Question and Answering (Q &A), Relationship Extraction, etc. In solving the above-mentioned problems, a pipeline needs to be built that follows a methodical workflow.

A typical NLP architecture is a pipeline of distinctive components that may start from either input speech or text data, followed by exploratory data analysis, pre-processing steps such as data cleaning, parsing, and feature engineering techniques whose purpose is to extract meaningful features that help in the task of prediction. There are various steps involved in a pipeline such as for text data—it involves segmentation, tokenization, lemmatization, stop words removal, dependency parsing, noun phrases, NER, etc. However, steps can be skipped or re-arranged depending on the NLP problem. Figure 3 shows a representation of components of a typical NLP system, starting from injecting natural language into the system.

Following that, the data passes through the natural language understanding stage, which performs various tasks of understanding the intent from speech, text, or both. In this stage, speech data may undergo transcription if necessary, otherwise known as speech-to-text (STT). Depending on the problem, deployed modelling and pattern mining produce outputs in this stage.

In the next stage, i.e. natural language generation, the output of the previous stage helps in generating a response with support from the back-end information source (service management databases, CRM systems, etc.). Following that, natural language communication helps in synthesizing a response into speech, otherwise called text-to-speech (TTS). Combining all the components results in a loop, which repeats each time new data is loaded into the system.

Sentiment analysis is to identify, extract, and quantify customers’ emotions and intentions, and translate them into data in real-time. Sentiment analysis tools have been widely used to analyse human feedback and monitor the level of satisfaction in various NLP applications, including social media content (e.g. [52, 53]) as well as in CCs platforms.

Earlier efforts focused on developing an integrated approach where CC data can be utilized for enabling business intelligence, text classification, and interactive text labelling for capturing customer satisfaction [54]. Later, [22] proposed a model that estimated customer satisfaction categorized as satisfied, neutral, and dissatisfied using a 5-point classification scheme, comprising of Naïve bayes, decision tree, support vector machines (SVMs), and logistic regression models. In relation to sentiment analysis, it has been widely studied and some studies have notably used CC data [49‐51, 55‐57]. In the last few years, sentiment analysis has gained major research interest, mainly because of its potential application in dialogue systems to produce sentiment-aware and considerate dialogues [58]. However, studies using real-life data extracted from CCs are scarce.

In the study conducted by [46], a method proposed predicted the emotional states (anger or neutral) of the users. Their method employs combining features with N-gram, sentiment words, and domain-specific words. Their study informs on ways in which features can be combined statistically to predict user sentiments. The result is enhanced user satisfaction in a call centre. The dataset that they used was of China mobile call centre. A combination of acoustic and linguistic rules applied supported the development of a multi-dimension model. The classifiers selected were SVMs, Maxent entropy, and traditional Bayesian. The main contribution of their work lies in how they incorporated the results from each of the individual classifiers they used in their work and added acoustic and language rules to it as well. An evaluation of experiments conducted highlighted that their fused system’s F1 measurement result improved to 69.1%, outperforming the baseline SVM model whose F1 measurement was 65.4% (Table 1).

Much attention has been directed to studying the emotional content using speech signals and many systems have been proposed. In [83], authors survey speech-led emotion classification which addresses three crucial aspects; suitable features for speech representation, design of a system, and preparation of a database. Numerous other works have also investigated the estimation of emotion classification and customer satisfaction at call level using acoustic features such as pitch, duration, energy, intensity, log frequency power coefficients (LFPC), and Mel-frequency cepstral coefficients (MFCCs) [22, 59, 62, 71‐73]. Subsequently, Bag-of-Words (BoW) and N-gram are also used in several studies to extract sentiment-related phrases [22, 61, 62, 64, 73, 77]. In the case of [77], features like call dominance or call–turn overlap that reflects customer emotions were exploited. In the work of [72], customer dialogue features like answer repetition were used. Historical events data on customer interactions and in-queue waiting or hold time found in the metadata of calls were used in the work of [22, 77]. SVMs have been mostly used in the above-mentioned works. In the study of [74], a method similar to call level has been utilized for emotion recognition, estimating customer satisfaction during the call using information from the start to the present call time. Features used at call level have proven to be effective [66] including call user’s gender as a feature [70]. Some studies have also proven the use of linguistic event features such as laughing to be also effective [60] as well as the use of visual features when it comes to video-based customer interactions [78]. A recent study in [81] proposed a framework for recognizing interlocutors’ emotions that are specifically designed for CC systems. This approach detects the emotional state of clients as well as agents using text and audio interactions. The study utilizes actual discussions that occurred during the operation of a big commercial CC. They used a wide range of NLP approaches including vectorization, word embedding, transcription methods, dictionaries of emotional expressions as well as multiple machine learning and deep learning classification methods for emotion detection. The detection accuracy obtained for the textual interactions was 70% for agent utterances and up to 60% for client utterances. Whereas, the detection accuracy obtained for the combined interactions (textual as well as audio) exceeds 68%. This method was utilized in [84] to develop an emotion detection method for CC conversations taking into account a wide range of emotions including, anger, fear, happiness, sadness, and neutral. The obtained results were in line with the previously achieved results for both textual and audio channels.

Since call-level customer satisfaction captures the global characteristics of calls, it often becomes too complex for it to work accurately on some real CC calls. For instance, some calls could contain both positive and negative customer reactions as the customer could be dissatisfied with the service at first and then might be either neutral or satisfied at the end of the call [48]. Another method where much attention has been given is an estimation of customer satisfaction and emotion recognition at turn level. Turn level can be explained as several unique segments by the speaker in a given call. It is detectable by identifying each customer turn from other turns between channels. Acoustic and linguistic features at the lexical level are most commonly applied in the turn-level task [49, 63, 65, 67‐69, 74, 78]

A study in [45] assessed the significance of acoustic features from customer-agent interactions to predict customer satisfaction using deep neural architecture. They investigated whether speech prosodic features can be complementary to speech transcriptions. Convolutional neural networks (CNNs) were trained on an amalgamation of acoustic features and word embedding for the binary classification task of “high” and “low” satisfaction. The real call centre dataset of a large Spanish corporation was used. A range of experiments conducted using various modelling approaches BoW, principal component analysis (PCA), XGBoost, and CNN were used. Their study first highlighted the point that linguistic features more accurately predict satisfaction than low-level prosodic and conversational descriptors such as fundamental frequency (F0), loudness and articulation rate. Secondly, turn-level features generally outperform call-level features. Lastly, on the application of fused linguistic and prosodic features using CNN, they reported the best performance of F-score 73.3% compared without prosodic which stood at 60.05%. Other similar works using CNNs also incorporate low-level acoustic features or Automatic Speech Recognizer (ASR) metadata as part of training data for their chosen models [75, 76]. In the study of [76] convolutional neural networks (CNNs) were used on audio frequencies to automatically learn valuable features and predict self-reported customer satisfaction from Spanish CC data.

Another study of [48] employs both turn and call-level features for estimating customer satisfaction. For turn level, they utilized prosodic, lexical and interactive features. They proposed a method that utilizes long-range sequential information and jointly optimizes them to assess the relationship between call–turn-level customer satisfaction. Long short-term memory recurrent neural networks (LSTM-RNNs) were used on call and turn levels to capture long-range sequential call contexts. Both were stacked hierarchically such that turn-level outputs can be utilized for call-level estimation directly. Three experiments highlighted that their proposed framework outperforms SVM and fully connected neural network (NN)-based classifiers for both turn level and call level. More recently, graph neural networks (GNN) was proposed to predict customer satisfaction in a real-life US corporate call centre that takes into account the relative satisfaction scores during training. Their experiments proved more accurate compared with standard regression or classification models [47].

The study from [80] used pre-trained Wav2vec 2.0 embeddings to detect emotions. The authors reported superior performance compared to the result in the literature for two open-source datasets. The authors proved that the Wav2vec 2.0 model performs better when Wav2vec features are combined with a set of prosodic features. Also, the work from [79] focused on a prominent research direction in representation learning, i.e. using pre-trained self-supervised learning (SSL) models as feature extractors to improve the task of emotion recognition. To achieve this, a transformer-based multimodal fusion mechanism was employed. Their results suggest that SSL features can be effectively used from pre-trained models and the SSL algorithms allow to leverage the potential within largely accessible unsupervised data. Upon evaluation, their approach outperforms the state-of-the-art models on four datasets.

Despite recent advancements in the automatic detection of customer satisfaction, it remains a challenging task due to the scarcity of labelled training data. Collecting large amounts of CC interaction data with customer satisfaction annotations is costly and time-consuming. Recently, authors in [82] have addressed this problem by proposing a customer satisfaction estimation method using unsupervised representation learning techniques. The method demonstrated its effectiveness using real-life CC data interactions.

Call routing also referred to as an automatic call distribution (ACD) can be explained as the process of placing live calls in a queue and distributing them to the relevant departments or agents based on pre-established rules and criteria as shown in Fig. 5. The rules can be based on both customer and agent behaviour, including common routing factors like the reason for the customer’s call or the amount of time an agent has gone without speaking to a caller. Intelligent call routing involving various routing strategies such as skills-based, longest available agent, and first available agent allows to instantly connect the caller to a specific phone line or extension without placing the caller on hold. Call routing impacts customer experience significantly as it can benefit in faster resolution, reduced wait time, decreased call abandonment rate, and a more balanced agent workload.

Several works have been published previously on routing calls using natural language call processing. Among many methods and approaches proposed were those using a boosting-based system [85], a vector-based information retrieval technique [86‐88], and a probabilistic model with salient phrases [89]. In [19], various CC functions are reviewed including call routing, skill-based routing, and networking. The authors outline important unaddressed problems and provide promising future research directions.

An article by [90] described a Markov queueing model with three groups of specialized agents and two customer classes. The authors believe that skills-based routing with priority-based rules produces both performance measures and steady-state probabilities. In the work of [86], a routing matrix was trained on statistics of word sequences and the occurrence of words in a training corpus following morphological and stop-word filtering. New user requests represented as feature vectors were routed based on the cosine similarity score with the model destination vectors encoded in the routing matrix. The performance of the above-explained routing system often depends on the routing matrix quality. In the work of [91], the use of discriminative training on the routing matrix was also proposed to improve accuracy and robustness. Instead of simply counting in conventional max likelihood training as shown in the work of [86], they use the min classification error (MCE) criterion in discriminative training of the routing matrix parameters. Discriminative training proved an effective technique when experiments were conducted, outperforming max likelihood classifiers by reducing error rate and increasing robustness. For evaluation, USAA call routing task consisting of 4000 calls belonging to a banking domain and QASIS task involving calls to the UK’s British Telecom (BT) operators were used.

Automating call routing has been a challenging task and complexity comes in combining several classifiers to optimize the process as well as when the process scales and involves many different classes (or decisions). This has been a complex problem that has only received little attention as discussed by [85] and [92]. The work of [93] provides a substantial solution to this problem by proposing a global optimization process based on an optimal channel communication model allowing for a combination of heterogeneous binary classifiers. The approach adopted was inspired by Markov modelling in which computational feasibility is achieved through simplifications and easy-to-interpret independent assumptions. The experiments showed call-type classification error rate decreased in a natural language dialogue system by 50%.

The discriminative term selection method has been explored in which the discriminative power of the term is measured. This is calculated by measuring the average entropy variation on the topic when the term is either absent or present. This helps in assigning a numeric value indicating its importance as shown in the work of [94]. The work from [95] highlights the benefits of improving a single classifier’s functionality by applying automated relevance feedback, boosting as well as discriminative training. The study aimed to construct a more accurate classifier. Their proposed algorithm performs by studying each iteration and using the one which is more accurate to minimize training errors. Results were compared to the baseline classifiers and 41–50% improvement in the classification error rate (CER) was observed. More importantly, synergised outputs of discriminative training on the boosting algorithm were also demonstrated and reduced the CER of re-weighted trained classifiers by an average of 72%.

A study from [96], experimented with four models—generalized linear model (GLM), NN, SVMs, random forest. Their study evaluated all four models’ performance and NN and SVMs were reported as better performers than the rest for the task routing calls. Similarly, the work from [97] used seven models to predict the most appropriate call operator for the customers. Their results highlight LightGBM as the best model and authors point out that using large amounts of business data can further improve the performance when using innovative algorithms. The work from [98] applied seven various term weighting techniques for feature selection tasks based on a self-adaptive genetic algorithm (GA). k-NN, linear SVM, and NN methods were used as classification models. Experiments demonstrated that the most effective term weighting is term relevance ratio (TRR) and the classification model is NN. Selecting features with self-adaptive GA proves highly effective for classification and dimensionality reduction.

In most natural language-based routing systems, the main purpose of an ASR is to transcribe a user’s request in a speech-to-text (STT) so that analysis on the transcription can be performed to determine the most appropriate service destination (agent). Given the level of uncertainty in accurately recognizing words by an ASR, the call can often be incorrectly transcribed, thus raising the possibility of calls being routed to the wrong agent. To tackle this issue, the study from [99] proposes a technique for using confidence scores that an ASR metadata contains to reweigh query vectors in a latent semantic indexing (LSI) classifier. Their results show that it can reduce the number of wrongly routed calls by a significant margin.

More recently, the study from [100] presents an intelligent call routing system that integrates text processing and speech processing. Their system route calls to the most suitable agent using routing rules built by the text classifier. It includes various components: telephone communication network, speech recognition, text classifier, and speech synthesizer. When evaluating the system in the real-world environment, the system proves its accuracy by achieving more than 95%. In call routing problems, understanding the context of customer requests or customer intention holds high importance and any context not understood well could potentially lead to problems. In a study conducted by [101], context analysis in call routing was investigated and an adaptive neuro-fuzzy inference system and HMM was proposed for solving this problem. Their system can be implemented in any language call routing domain since there are no syntactic or lexical features used in the classification task. Their proposed system reduces errors and increases accuracy to 93% on their dataset.

Yang et al. [102] proposed an automated call routing system that monitors all active live chat conversations in real-time to identify unsatisfied clients who wish to escalate their issues before they end their calls. The intention is to automatically direct their calls to a specialized agent who can help them address their issue before they end the interaction with the original agent. They use a hybrid model by integrating recurrent neural networks with manually engineered features. Experiments show that this method outperforms competitive baselines improving customer service.

The work from [103] proposed an automated triage design that reduces transfer rates and improves routing accuracy in a live chat using combined results from five ML algorithms (SVM, neural network, random forest, Naïve bayes, and adaptive boosting) and text analytics. For evaluation, a real-world large-scale dataset was used and it is noted that routing performance improved by 14%. However, many possible real-world scenarios such as customers with multiple questions that are handled by different CC service categories were ignored as stated by the authors (Table 2).

Several works have been completed on analysing customer interactions data that help automate different CC tasks. For instance, areas where customer interactions data has been analysed, include call-type classification for categorizing calls [104], acquiring call logs summaries [105], monitoring and assisting CC agents [28, 106], and development of domain models [107]. Identifying and filtering controversial dialogs from the automatic speech recognizer has also been explored [108‐110].

Another area well studied is insight mining patterns in databases where associations are made through structured dimensions [111]. For textual data, many ML-based approaches to mining and classification have been studied [112, 113]. In the research of [114], a method has been proposed to automate the process of extracting knowledge from emails. Their paper reviewed four generations of building systems and their challenges. Their approach used NLP techniques and the results were encouraging; however, they argue user intervention is still required for the system to be accurate enough in providing substantial results. Topic unigram language model has also been explored on counting the word occurrences for each topic as well as storing all words for each topic. The probability of the query in every topic is calculated and the optimal and most resembling is selected [115, 116]. The study performed [117] an analysis on agent entered call summaries of customers by extracting words based on domain-specified standpoint. In another analysis, insights were extracted based on the usage frequency of the dialogue patterns within customer interactions [118] and [119] analysed and attempted mining from a collection of complete interactions (recorded calls data) from a rental car reservation office to predict whether a customer intends of making a booking or not. Their work identified accurate standpoints and nominated expressions for every standpoint, thus resulting in the chance discovery of valuable insights.

Alternatively, the study from [28] proposed a system that automatically analyses a large number of CC conversations to provide an interface to CC managers measuring CC agent performance. Similarly, the study of [120] assessed the performance of call centre agents like time management or quality by adopting a variety of decision trees, neural networks, and statistical techniques. Also, the study from [121] developed a continuous-time Markov chain model that optimizes the call centre queuing process, thus promising to reduce hold time.

A recent call summarisation study for CC platforms was proposed in [122]. The study applies and compares the summarisation performance of various extractive summarisation methods. These techniques work by selecting key/important sentences from a given text and present them in the summary verbatim. Unlike abstractive summarisation techniques, extractive summarisation tools are unsupervised methods; hence, they are easy to develop and deploy as they do not require labelled data for training. The paper conducted a comparative analysis of such methods by comparing the summarisation performances of CC calls using subjective and objective evaluation measures. The study reveals that TopicSum and Lead-N methods outperform the baseline summarisation methods as they can produce meaningful summaries of CC interactions.

Although text and audio mining of call centre data have been researched, sequential analysis of the same has not been thoroughly explored. Sequential models have distinct applications but rarely do they appear to be focused on business intelligence. Their most common applications are within telecommunication systems, game strategies, inventory management, and maintenance problems as discussed by [123]. The model proves effective for decisions where outputs are partially controlled and random, thus helping to depict problems and compare strategies objectively. The study from [120] and [121] although adopt sequential techniques they focus precisely on staffing instead of an evaluation of CSR strategies that facilitate conversational flow and outputs. The study from [124] adopted distributed computing in the development of topic models from call centre conversations. Although the NLP technique used produced high-level insights, it did not help identify the sequential insights and proved insufficient for turn-level process improvement. In contrast, the work from [125] took into account the sequential nature of agent–customer conversations and used a Markov decision process (MDP) to identify customer states and agent actions. This helped them to identify the most frequent sequence from successful conversations and estimate outcomes when an agent performs a particular action for a customer in that given state. This helps in process improvement and training agents as ideal outcomes can be often used to direct customer conversation flow such that it concludes positively, thereby providing an overall better experience to customers.

Concerning call-type classification, the work from [126] put forward a method enabling automatic identification of calls that were problematic and required managerial evaluation for call centres. In the work of [106], a call centre monitoring system was proposed which facilitates text analytics and information gathering. Their system analysed the content of call centre data and detected the main issues pointed out in the data. In [110], a system was presented which could recognize speech and apply text-mining techniques for French call centre data. Whereas, the work of [126] shows an interactive mining tool built on pragmatic analysis and applied to a data corpus containing manually transcribed call centre interactions within the banking domain. Meanwhile, the author mentioned the limitations of the transcription process as not accurate and incapable of identifying phrases that accompany emotions such as gratitude or sarcasm (Table 3).

Another area of research interest in the domain of CC has been the use of chatbots or virtual agents and speech-enabled IVRs. Chatbots are essentially part of a system with dedicated components such as a dialogue manager, responsible for communicative goals, which is interfaced with a task manager that knows the underlying goals of the communication. Regardless, both are responsible for natural language generation to produce meaningful language utterances which fit the circumstances and specific goals are achieved by following appropriate courses of exchanges. Such a system is often part of a large spoken dialogue system as well such as speech-enabled IVRs in CCs. The workflow of a typical chatbot is illustrated in Fig. 6.

In an early study of [127], technical innovation within AT &T’s eContact space focused on voice-enabled CC automation highlights VoiceTone, an intelligent virtual agent that uses speech and language technology. It acts as a replacement for an existing IVR system and converses naturally to complete customer requests. It emphasizes replacing a cumbersome, menu-based interaction with a more natural and flexible user experience. For the development of a conversational agent, the MDP framework has often been applied. Another early study in [128] proposed a learning dialogue system that used stochastic MDP for an Airlines information system. While the model could successfully reveal optimal strategies, it was not used on the human-human dialogue system but a man–machine system that has less variability than the former.

Over the last ten years, there has been a growing interest around chatbots in CC systems (e.g. [129, 130]). Chatbot technologies gained further attention following the COVID-19 pandemic, which transformed the model of interpersonal communication. A chatbot implementation in [131] was proposed to improve virtual communication with people and provide them with answers about the COVID-19 disease. Another recent work in [132] developed chatbot tool to help with the daily screening of healthcare workers to prevent the spread of COVID-19 in the healthcare setting.

One of the key challenges in modern chatbot systems is to design accurate automatic models for customer intent detection. Early work in [133] proposed a hidden Markov model (HMM) system to model the intention of a sentence using the Viterbi algorithm. The model not only considered the phrase frequency but the syntactic and semantic structure of a phrase frequency. It is substantiated that an accurate determination of the caller’s intention helps significantly in conversing functionality. The experiment results showed a correct response rate of 80.3%. A method that combines two different approaches (Hidden Markov and neuro-fuzzy models) has also been suggested which automatically identifies user intention in a dialogue. The results show that the overall performance of a human–computer dialogue system improved [134]. Other approaches have also been suggested [135‐137]. The work from [138] surveys several past and present computational approaches to natural language that generate utterances by using speech acts or words as particular types of actions in solving a problem.

In contrast to other approaches, reinforcement learning (RL) is suited particularly for such tasks where the best strategy to achieve a goal is unknown and the system tries to automatically find an optimal policy from interactions with the user and the environment. An interesting study is from [139] in which hierarchical reinforcement learning (HRL) is used for jointly optimizing spatial behaviours and dialogue behaviours. The proposed method learns to provide navigation instructions by making use of the customer’s prior knowledge into account. To improve AHT or response times, CCs need to build systems that can categorize user requests, complaints, and questions and filter them by priority keywords. Also, an automated process that works like a search engine and recommends possible solutions to CC agents. The automated process must have the capability to surface content quickly and offer insights by identifying the relevant patterns from the data. One such publication presents a novel approach in which HRL is utilized for natural language generation in a dialogue system that learns the optimal utterance through reward function [140]. The proposed method optimizes content selection, utterance planning, and surface realization decisions in a joint fashion, otherwise strictly interdependent. Results show that their combined approach outperforms baselines that followed the independent optimization approach. More recently, [141] conducted a study in which a Markov process describing a model function was constructed. The numerical assessment of their model highlights a positive effect of chatbot usage particularly when CC is experiencing an overload of customer queries.

Modern CC systems are increasingly using intent recognition systems in their chatbots systems to improve the quality of their virtual assistance. Recent studies have focused more on this direction by proposing more accurate and robust models for recognizing customer intent. For example, [129] proposed an intent recognition system in CC platforms that takes into account certain human emotions in customer-agent interaction. They used inference rules to detect human emotions regarding the actual intentions of the customer using recorded CC calls given in the Polish language. Another work in [142] introduced an evidence-based machine learning framework for the automatic detection of subjective calls. They used deep neural network to assess a corpus of seven hours of recorded calls from a real-estate CC and achieved an accuracy of 75% for subjectivity detection (Table 4).

A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations (MELD)—an enhanced and extended EmotionLines dataset has been selected for this experiment [143, 144]. MELD contains similar dialogue instances that are available in EmotionLines, but it encompasses audio and visual modality along with text also. MELD contains about 13,000 utterances from 1,400 dialogues from the TV series called ’Friends’. The textual part of the dataset included two label columns. The column ’Emotion’ contained seven labels: neutral, joy, sadness, anger, surprise, fear, and disgust labels. The column ’Sentiment’ contained three labels (positive, negative, and neutral) which are what were used in our experimental study. The audio part of the dataset was retrieved from converting MPEG-4 Part 14 files into a WAVE format. Our experiments used 9988, 1108, and 2608 audio files and textual utterances for training, development, and testing. The data was passed as a CSV file with columns for text and sentiment label and an audio file directory path. The statistics of the MELD dataset are presented in Table 5.

Different models were experimented individually on two different data formats, i.e. audio and text. For audio—2D CNN and Wav2vec models were experimented, whereas for text—ALBERT, BERT, and RoBERTa models were experimented with. This eventually helped in comparing the model results of the MELD dataset and shortlisting the best-performing model for the fusion experiment. For audio data, 2D CNN was trained and deployed with MFCC and spectrogram features. The outperforming model was the one trained on Wav2vec 2.0 large, followed by Wav2vec 2.0 base and 2D CNN MFCC as shown in Table 6. For text, RoBERTa’s performance was notably better than the rest of the models as shown in Table 7. The RoBERTa model performed even better when the output of its last four hidden layers was concatenated and used for predictions as opposed to using the last layer only, i.e. pooler output. Following individual model training and deployment, the audio and text embeddings from best-performing models were fused and then loaded into the RoBERTa model. Despite our initial expectations, the fused embeddings (audio and text) loaded RoBERTa model did not exhibit any improvement over the text-only RoBERTa model, as demonstrated by the results in 8. We evaluated all models using multiple metrics such as weighted accuracy, loss, and F1-score. The number of training epochs used was set to 10, the learning rate was set to 2e-5 and the batch size was set to 16 for pre-trained models—text input and 20 training epochs, 0.001 learning rate, and 16 batch size for 2D CNN and Wav2vec 2.0—audio input.

The primary objective of this experiment was to showcase the advantages of using pre-trained transformer models for sentiment analysis. We first used audio and text data separately and then combined their features to evaluate the performance of different models as shown in Table 9. The results presented in this study clearly demonstrate the potential of utilizing the latest NLP techniques to achieve better results. This experimental study could serve as a guiding framework for developing sentiment analysis systems in the future. It can also help CC organizations in driving innovation by leveraging the latest models. However, it should be noted that this experiment is a proof of concept, and more research is needed to develop a production-ready sentiment analysis system.

The results of this study indicate that transformer models perform better than classical ML and language models, particularly for textual data. Hence, we recommend the use of transformer models for sentiment analysis tasks. However, the performance of the audio data was not satisfactory, indicating a need to explore a wider range of features in future studies. Some of the limitations of this experiment include the small dataset size, transcription quality, and lack of better audio-quality data. Future studies could focus on addressing these limitations to further improve the accuracy of sentiment analysis systems. Also, the focus should be given to implementing newer advanced transformer-based language models. Overall, this study has shown the potential of transformer models in the field of sentiment analysis and offers valuable insights for future research.