Counseling (ro)bot as a use case for 5G/6G

We proposed and developed a software robot (bot) called VICA: VIsual Counseling Agent [4]. It is a distributed AI (Artificial Intelligent) application for medical services such as mental healthcare to assist emotionally distressed persons. It does a kind of remote mental operation, although not physical. To cope with emergencies such as pandemics, the original system that we call VICA (basic) is gradually extended in consultancy function when questioned for anxiety reduction. It is also going to recognize the emotion, body language, or physical condition of clients (distressed persons) from images of facial expressions and other sensors such as body temperature, etc. This extended VICA is called VICA advanced. In this section, VICA basic is explained as its foundation.

VICA uses CRECA [2] as a baseline system. CRECA is a text-based system composed as an AI counseling server of VICA. The architecture of CRECA is shown in Fig. 1. Terms and their structure are extracted by standard Natural Language Processing techniques with a dictionary for emotional words and then saved in context objects. The agent creates its dialogue responses using information stored in current context objects and its counseling knowledge database (DB).

CRECA uses not only ELIZA-style mirroring but also generates its own responses for digging [5]. Furthermore, it periodically attempts to summarize the clients’ previous utterances considering their emotional impact and chronological order. Asking about the quality of summarization, the agent can check how accurate the listening has been. This also helps clients in seeing their own distortions or reviewing their thinking. Finally, the agent prevents the dialogue from drifting away from the problem even when the client’s emotional state has changed. This preserves the client’s trust in CRECA and keeps the dialogue open.

Sentence patterns used in this phase include standard problem-digging phrases (e.g., “please tell me more” or “give me more details”). Such digging responses give the client a chance for self-reflection.

Different from CRECA, VICA has a voice (speech) and visual interface as its input/output terminals or clients of client–server systems. VICA (or more concretely VICA’s CRECA) is starting to have the function of consultancy or answering to reduce clients’ anxiety. Anyway, it is important for CRECA, and therefore VICA, to continue the conversation as well as recognize a client’s emotional state. Besides the client’s emotional state recognition error, the error in speech recognition is a significant problem.

VICA’s processing flow is as follows: CRECA performs the rule-based analysis of input sentences by pattern matching and the creation of a response/log for paraphrasing, digging, and summarization.

Client’s input: For me, software developer is a wish in life, but lack of programming skill is an obstacle. Job hunting makes me feel worried.

VICA’s reply: Hum, Hum, for you, software development is a wish in life, but lack of programming skills is an obstacle. Job hunting makes you feel worried, doesn’t it? Please speak a little more in detail.

Client’s input: I am always the last student that finishes given program problems in my programming class.

VICA’s reply: Hum, Hum, you are always the last student that finishes given program problems in your programming class. Please say more about it.

VICA’s reply: Hum, Hum, you have no skill of rapid programming. Please tell me the relation among lack of programming skill and software developer.

Client’s input: Oh, I found I can develop good quality programs and I would like to do research and development of software

VICA’s reply: Hum, Hum, you found you can develop good quality programs and you would like to do research development of software, wouldn’t you. Please tell me more.

Client’s input: Now I am fine.

Below are network-related techniques used in VICA [4]:

However, in the current 4G networks, there are problems as follows: (1) It has an error every two sentences or so⁴. (2) Some words drop or disappear during recognition. (3) It sometimes takes more than 10 s in response time, or in the worst case, the connection/task fails altogether. Thus, in 4G networks without slicing, the cloud-native speech recognition of VICA has a reliability problem. It is a significant problem, because VICA uses “sensitive listening” skills as its counseling methodology [7]. The skill requires sensing or listening to clients’ minds through recognizing clients’ sayings and their very sensitive emotions with extreme accuracy. Then, it continues conversation or communication to promote clients’ reflection towards their self-awareness. Thus, VICA has a serious reliability problem in such speech recognition of AI cloud services. Even though accurate image recognition may help to solve the problem, accurate emotion detection/recognition from such as facial expression is further difficult due to a large amount of image data transmission burden in networks. Such reliability problems of AI recognition Cloud services have to be solved, so that our AI counselor “VICA” can be practical in real use.

It is important that VICA further guarantee or assure speech recognition accuracy/reliability in Google speech-to-text cloud service. To attain this, also using the distributed computing approach, VICA servers more densely collaborate with its clients/terminals. In this case, VICA has another problem of transmission over inter-carriers in the reliability and/or latency of networks if terminals are located far from servers as those in foreign countries. VICA has clients for speech I/O (Input/Output) and servers for AI counseling. Clients composed of PCs (Personal Computers) or mobiles handle speech/image recognition to obtain the input from patients. Servers mainly create responses for counseling from the input. However, for VICA using “sensitive listening” type counseling methods, speech recognition quality is fatal. VICA servers must assure it before creating the response for counseling. It is done as follows. (1) VICA terminals send voice data to AI speech recognition Cloud servers in such as Google Clouds. (2) VICA terminals receive the recognized speech text. (3) The text is sent to VICA servers. (4) VICA servers validate the resultant text by perplexity checking, etc. (5) On suspecting the error, they improve by several methods before CRECA generates a response for AI counseling. For example, these methods are various kinds of n-gram (forward, backward and so on) with weighted majority decision [11]. (6) If the error is still suspected, servers send text messages to ask clients (terminals) to redo the speech recognition using Google Cloud. (7) To attain high accuracy, this loop should be repeated until the error is not suspected. Meanwhile, the number of loops is limited, so that the loop stops when elapsed time becomes more than the specified time. The elapsed time is defined as t\(_{c}\) (current time)-t1 (input starting time of the current dialogue). The specified time can be adjusted. However, the maximum time is also limited, since too long dialogue response time very often stops the conversation. Thus, the network performance affects the number of loops, which affects the quality of speech recognition service. Thus, an ultra-reliable low-latency network such as that of 5G with slicing is required. The next section shows these network requirements.

During emergencies, extraordinarily many persons will use counseling AI systems at the same time. For example, in large cities, around 70K possibly more than 100k persons are expected to use VICA concurrently [4]. Thus, not only reliability or latency but data rate (bandwidth, transmission speed) becomes problems during emergencies or a surge in demand.

To solve problems or to realize expectations mentioned in the previous section, VICA requires no disconnection, less than one word every ten or more sentences of recognition error, and less than 10 s of total response time. This is indispensable for distressed/anxious persons (clients) to continue the conversation or virtual meeting via voice or further including image and other sensors towards self-awareness.

Thus, the network requirements are considered as follows: Requirement details for delay are as follows:

Currently, experiments show that 5G is not sufficient for even just AR (Augmented Reality) or AD (Autonomous Driving). Meanwhile, Counseling is a kind of mental operation. Even one word of slow, missing, or incorrect recognition of the client’s voice can easily stop the conversation. It stops the counseling for mentally distressed persons or the consultation during an emergency. Counseling (consultant) AI requires quality communication such as less than 1 recognition error per 100 sentences or 1000 words. It is 10 times that of excellent speech recognizer’s quality. For iPhone or google assistant, it is 1 error per almost one sentence (much less than two sentences) or per 20 words or Japanese characters⁴. Even the high-quality Google speech recognizer makes speech recognition (text conversion) error of around every two sentences. It sometimes causes more than a few seconds delay on the cloud. Furthermore, there is also an encryption overhead for secure (privacy-preserving) communication that is necessary for counseling or consultation.

Consequently, in speech conversation, even the Cloud Google speech recognizer has errors.

Such requirements detail for data rate are as follows:

Counseling/consultant AI (VICA) requirement for Voice (including tone, 5 majority) is upstream over 10–20 (− 100) Gbps. For example, during the Coronavirus emergency, at least one prefecture had more than 1K counseling calls/day even before its explosion, simultaneously 1K calls maximum possibly. Each voice call requires 1.5M (=64 Kbps*24 channel) bps [13]. 64 Kbps can be compressed to 32 Kbps in no loss of sound quality and 1.5 Mbps becomes 0.75 Mbps. However, the majority decision out of 2-5 cloud speech recognitions makes it more than 1.5 Mbps again. There are also problems with surrounding sounds and echoes for long distances.

Besides AI counseling calls, there are basically Internet telephone (inclusive of human counseling or just asking information, etc.) especially after school or office. Tokyo population is 14 M. On Corona explosion, if 7–14(− 70) K persons (e.g.) expected to have such calls at once during several minutes, upstream will require (1.5 Mbps * 7M =) 10–20–100 Gbps in Tokyo. It takes much more if the canceling of noise such as echo, surrounding sound, etc. is considered as well as compression with high quality maintained even on various tones such as rising/faint tones. This is because speech recognition mistakes of even one word can stop counseling. It is still worse if the connection/task fails in the worst case.

For realizing the very accurate/stable recognition required by “sensitive listening” of VICA, the next section proposes the method to fulfill and assure these requirements, using 5G/6G slicing inclusive of Cloud.

The proposed solution is a cloud-native distributed system architecture with a service catalogue providing the multiple levels of the menu to assure the service level by such as SLA (Service Level Agreement). This stably enables high performance as well as an optimal cost network. Towards 5G/6G use case PoC of VICA, it aims at preventing cloud speech recognition from frequent errors (e.g., every short sentence) and significant response problems/delay or disconnection in the worst case. The flexible network configuration is illustrated in Fig. 3 where External Network-to-Network Interface (ENNI) is also presented. The service catalogue is proposed to cope with congestion during emergencies, disasters, etc. For instance, users at the VICA terminal can select a menu from the following 4 levels of the service catalogue: In such a model, the service-level catalogue for quality counseling needs to be more than level 2 to maintain speech recognition quality. The cost is paid by health insurance, hospitals, or the user’s subscription fee. In an emergency, ministers can order automatic 1–2 level upgrades of catalogues, whose cost is paid by the national tax. Such catalogue is shown by VICA terminals. Users (clients or patients) select the service level (1–4) in the catalogue menu. VICA terminals send it to AI counselor server (VICA server). VICA server reports to manage it as a basic requirement to EA (Enterprise Application). EA reports the basic requirement to CRM (Customer Resource Manager). Through the report, NaaS (Network as a Service) via BSS (Business Support System) of CSP (Communication Service Provider), etc. assigns network/Cloud resources including MEC (Fig. 3) to fulfill the basic requirement according to an Service Level Agreement (SLA) and the like.⁵ Meanwhile, the VICA terminal monitors the delay and sends it to the VICA server. VICA server logs the overall delaying situation of each service level to analyze the problems for VICA 5G use case PoC. VICA server reports it also to EA /CRM. As shown in Sect. 4.2, through the report, NaaS via BSS can perform optimal network operation to assure the basic requirement.

This section proposes the assurance method for VICA 5G use case PoC. Assurance/re-fulfillment operation is important for dynamically solving the above-mentioned VICA’s problem in the AI speech recognition cloud.

As described in the previous session, the base requirements of the service-level catalogue are initially selected by users (clients), set by AI counselor terminals (VICA terminals), and fulfilled by NaaS of CSP, etc. However, during counseling sessions also, they are kept assured dynamically by NaaS as follows: AI counselor terminals (VICA terminals) monitor the delay during a counseling session and send it to AI counselor servers (VICA servers). VICA servers dynamically report the overall delaying situation of each emergent level to OSS (Operation Support Systems)/BSS of NaaS. It is done by network operation API such as Skynet in the EA domain. On the detection of serious delay, NaaS automatically performs the network operation such as slicing to assure or, in case of changing the network configuration re-fulfill the basic requirement dynamically in real-time.

The next section proposes a way to further improve the reliability of VICA using AI recognition cloud service inclusive of AI speech recognition as well as AI facial expression detection. To implement such a solution for improving speech recognition accuracy a counseling agent VICA is extended as advanced VICA in its distributed system architecture. It also proposes a way to estimate the effect quantitatively, especially for 5G use case PoC. This helps to show the usefulness of 5G to VICA and its problems causing the necessity of 6G.

Network resource fulfillment is done according to the basic requirements as mentioned in the previous section. Namely, network configuration or slicing is set initially when users selected the service level (catalogue menu) of the AI counseling application VICA. Depending on the social situation including the number of patients and the minister’s public requests on the pandemic, the slicing (network resource allocation) is done dynamically and automatically. It can be done proactively beforehand though possibly in 5G/6G or in the future. This is because it takes too much time to finish such slicing especially when it is done manually. More dynamically, the AI terminal of VICA monitors the delay in real time and sends it to the AI server of VICA. The VICA server at the EA domain reports the overall delaying situation of each emergent level to service providers (CSPs). Through the delaying information, NaaS of CSP can change the network resource or its configuration to assure the basic requirement via CRM and BSS/OSS.

Furthermore, at this point, 5G facilities are limited or very expensive. Both 5G and automatic network operations are not available for our experiments. To cope with this problem, our 5G validation experiments for 5G use case PoC of VICA simulate the effect of network (slice) operation by substituting the cloud (remote) server with the local server.

Figure 4 shows this problem and solution approach for 5G PoC of VICA use case as follows: Problem: At this moment, not 5G but 4G is used in a city in Japan. Even text transmission delay among VICA client (VICA terminal) and VICA server significantly decrease the effect of 5G (not including transport slice) use if the AI server (VICA server) is set in the city in Japan. Approach: AI server is set also in European/American communication provider’s edge or core network to simulate 5G slice effect of less latency in edge compared with less latency effect in inter-carrier.

Method: Emergency base prioritized implicit network operation effect for 5G (shown by delay data at AI terminal) to Google Speech Recognition Cloud as well as to AI counselor server 1 set in Europe/US is compared with the effect to AI counselor VICA server 2 set in a big city in Japan. Just VICA server 1 in Europe or the US sends initial base requirement of emergency level (catalogue menu) selection and dynamically (during counseling) monitored response time to CRM/NaaS through network operation API such as Skynet for 5G slicing.

This section describes the network architecture and evaluation method for 5G/6G use case PoC of enhanced VICA. Figure 5 shows the architecture of enhanced VICA. Using AI speech recognition cloud, AI counseling Terminal (VICA terminal) obtains text as the speech recognition result. However, in the showcased use case scenario. the text is erroneous or missing as mentioned before.

Communicating with VICA client, AI counseling server (VICA server) collaboratively improve the result as follows [11]:

Nowadays, people in foreign countries need counselors in their mother country. In such cases, the server of VICA is connected with VICA terminals over inter-carrier networks such as between Europe/the US and Japan. In the case of the VICA server collaborating with VICA terminals for improving speech recognition reliability, the communication is repeated many times if errors continue. In an emergency, such communication over inter-carrier networks increases sharply. To decrease the delay or burden over inter-carrier networks in such cases, VICA enhanced versions are expected to make use of 5G/slice.

For the 5G/6G use case, PoC of VICA, network operation, and its validation to improve cloud speech recognition reliability are described in two cases of VICA server deployment. For instance, as Fig. 6 shows, VICA server1 is deployed in Milan in Europe and VICA server2 is in Japan. In these two cases of deployment, VICA enhanced version performs quality assurance through network operation API for 5G/slice to decrease the burden over inter-carrier as follows:

Emotionally distressed persons/clients do not verbalize their emotions very much. Even if they say it, there is a good chance that their utterance becomes faint or unclear. Sometimes, even humans cannot catch their voice. In addition, speech recognizers may fail to recognize the voice. Especially, for “sensitive listening” type counseling advocated by Rogers [7] which VICA adopts, it is important to know clients’ emotions or their change very accurately without intrusion or without asking them explicitly. Human counselors know client’s emotions through their facial expressions or physical behavior. Meanwhile, advanced VICA tries to do this through the fusion of sensor/image data, incorporating real-time closed-loop feedback. To produce emotional words that VICA uses for the creation of counseling responses, the result is integrated with one obtained from the speech recognition of the user utterance.

Facial expression recognition is used to evaluate facial cues to interpret and understand the emotional state of patients, including emotions such as anger, fear, happy, or sad to assist in providing proper assistance. As to this, we have the first version of facial expression detection for mental stress. Sad feelings to indicate anxiety and happy feelings to indicate “clear up” or “fine” (“solved” or “self-aware”) are detected though a bit too simplified as the beginning attempt. Defining the former as a negative feeling and the latter as a positive feeling, for example, the change is detected in the feeling during the consultation. The change of emotion is very important for VICA to let patients reflect towards self-awareness by saying “You initially said xxx but now you are saying yyy”. Facial expression detection AI integrated into VICA can learn automatically as well as incrementally by logging the pairs of words (user utterances) and the corresponding facial expression. Using such incrementally learnt/trained face expression detection AI, VICA can recognize emotional change even if patients do not say any emotion words or say them unclearly [14‐17]. If patients do say emotional words, the accuracy of such emotional change recognition is improved by majority decision, etc. As to the improvement by majority decision, extracting continuous facial data to analyze essential features captures vital health signs such as stress level, fitness level, blood temperature, and blood pressure. Integrating this information also, advanced VICA can have another way of emotion recognition in near real time, which contributes to more accurate emotion change detection by majority decision. With the sophisticated algorithm and cloud-nature supreme processing power, the possibility is limitless. These improve the performance (total accuracy or reliability) of VICA.

Recognized emotion can have more degrees such as very sad (very negative), neutral (neither sad nor happy) or a bit happy, and very happy (very negative). Anxiety degree for advanced VICA is defined as such in terms of facial expression measurement. It is integrated with emotion detection by voice/speech recognition. This allows it to make near-real-time judgments and selection of words for communication with the patient. Thus, whole expression logic is integrated with speech recognition. This collaboration is indispensable to enhance the total accuracy of emotion change recognition for counseling or consultancy providing vital health advice as required by “sensitive listening” type VICA.

The advanced solution for a complex scenario where a patient seems smiling, but is still under stress will be implemented soon (e.g., movement of lips, movement of eyebrows, movement of the chin, movement of chicks, eyes movement, dialogue delivery, head movement, heart rate, blood pressure, and oxygen level). Of course, this will be combined with analysis of sound output, dialogue delivery, etc. as part of the audio analysis.

Eventually, combining all such recognition/analysis of various/large volumes of data, we can realize medical grade or very accurate sensor fusion necessary for AI counseling. Due to such ultra-reliable high-performance transmission/analysis of large data, VICA is further considered as use case of 5G/6G.

As shown in Fig. 7, for 5G/6G use case PoC of VICA, we propose two types of server deployment for counseling. The one is an advanced VICA equipped with AI servers for sensor/image fusion. The other is the human cooperation version of VICA equipped with AR servers for remote human counselors cooperation. These include image data transmission as follows:

Full AI counselor/consultant: body language including facial expression is recognized by AI AR servers are set independently as well as remotely for remote human counseling.

The system architecture of the VICA advanced version is as shown in Fig. 8. Compared with VICA basic version, besides voice data, a terminal of the VICA advanced version obtains various kinds of sensor data such as facial expression images from emotionally distressed persons. It sends them to the cloud-native AI recognition servers for analysis. It receives textual signs indicating the client’s emotion or condition. These textual signs and client’s utterances are sent to the server of the VICA advanced version. The server of the VICA advanced version (VICA server) integrates/fuses this information. If errors are suspected, such recognition (including speech) and integration are repeated for ultra-reliable (very accurate and stable) “sensitive listening”.

Cases including congestion-period, 4G, local server, and no slice were measured for experimental evaluation of VICA. More specifically, the conditions were as follows: congestion period (4 pm-11 pm), 4G and no slice from AI terminal to Google cloud (AI terminal is in Tokyo Metropolitan Area), but ultra-speed network simulating 5G/6G connection among AI sever and AI terminal by setting local AI server.

Results are shown in Tables 1 and 2 and can be interpreted as follows: For the AI terminal, the experiment utilized Windows 10 Home machine having Intel Core i7-5500 U CPU, 4 GB RAM, and 453 GB HDD, while for the AI server, Windows 10 Pro machine having Intel Core i7-7700 CPU, 32 GB RAM, and 953 GB HDD was used. Four Japanese nationals were used as human test subjects: a 30 year old female IT worker, a 50 year old male IT chief engineer, a 50 year old male retiree, and a 70 year old IT scholar. They were given the paper containing Japanese sentences to speak to VICA for counseling and the paper to write the test results. The contents of the sentences are shown in Table 4. The length/number of the interchange (dialogue/conversation) is ten. Each interchange has 1–2 sentences. They talked to VICA according to the given sentences. The test was repeated ten times per subject. Subjects recorded the number of VICA’s speech recognition errors (wrong recognition of each word) as well as disconnection or no response from VICA.

As shown in Table 1, using 4G networks during the congestion period, even just speech recognition time t2 – t1 was more than 5 s, which occurred three times out of seven times. The total response time t3– t1 for one dialogue was more than 10 s in 2 of 7 cases. These never satisfy the service requirement for response time as is described in Sect. 4.

As shown in Table 2, VICA (basic version) sometimes failed in Google cloud speech-to-text service. It happened in case of unclear pronunciation or stumbled/clogged wording. For various persons, it did not happen in the local service of Google speech-to-text without using the network, though it is not dialogue/conversation just commanding or dictation. Indeed, in this case, the speech recognition error occurred but complete failure did not. Meanwhile, a speech recognition error in dialogue that sometimes can stop conversation continuity occurred. It is fatal in counseling. The counseling server communicating with the client should detect and recover it semantically before creating the response.

As shown in Table 3, speech recognition errors in Clouds became more than double when compared with those in Edge. (Namely, Clouds 25 versus Edge 12 in dropped/disappeared sentences or phrases during recognition and Clouds 38 versus Edge 10 in recognition error to wrong words). Here, Google speech-to-text Clouds was used as the Cloud, and speech recognizer of Google Pixel 3A mobile was used as the Edge. The results were obtained using 11 Japanese sentences that are difficult to recognize. These examples are shown in Table 4. These example sentences are repeated in three kinds of speed such as normal, a bit slow, and slow. VICA, using the Cloud, had too many errors in the very fast utterance of such long sentences to continue the experiment. However, the Edge could recognize.

These results indicate that the proposed catalogue-based quality assurance mechanism using 5G/6G slice inclusive of MEC is necessary as well as expected useful for VICA. The reason is clarified as follows:

It causes a serious negative impact when clients become too irritated to continue counseling especially in case of emergencies on pandemics such as COVID-19. Thus, the final user effects on our use case VICA were evaluated mostly during the network busy period. Namely, the following user-level (i.e., application level) questions were posed to five distressed persons/patients (clients): Each item has the following selections: Type A. problems; Type B. small problems; Type C. no problems. As shown in Table 4, the results were as follows: Thus, we had 80 % of the subjects who felt the problem by network delay inclusive of the network-related failure especially depending on speech timing as well as by speech-to-text translation inaccuracy in 4G. Furthermore, merely unclear voice caused the disconnection with speech-to-text cloud servers as Table 2 also showed. The word errors when saying long sentences easily happened even when speech-to-text cloud servers were disconnected. These also irritated subjects. These qualitative evaluation results showed that 5G/6G including slice is necessary.