Persistent progress in the self-management of their disease is important and challenging for children with diabetes. The European ALIZ-e project developed and tested a set of core functions for a social robot that may help to establish such progress. These functions were studied in different set-ups and with different groups of children (e.g. classmates at a school, or participants of a diabetes camp). This paper takes the lessons learned from these studies to design a general scenario for educational and enjoying child–robot activities during returning hospital visits. The resulting scenario entailed three sessions, each lasting almost one hour, with three educational child–robot activities (quiz, sorting game and video watching), two intervening child–robot interactions (small talk and walking), and specific tests to assess the children and their experiences. Seventeen children (age 6–10) participated in the evaluation of this scenario, which provided new insights of the combined social robot support in the real environment. Overall, the children, but also their parents and formal caregivers, showed positive experiences. Children enjoyed the variety of activities, built a relationship with the robot and had a small knowledge gain. Parents and hospital staff pointed out that the robot had positive effects on child’s mood and openness, which may be helpful for self-management. Based on the evaluation results, we derived five user profiles for further personalization of the robot, and general requirements for mediating the support of parents and caregivers.
1 Introduction
1.1 Diabetes Type 1
The growing burden of chronic illness on health and health care has globally led to health policy responses increasingly referring to self-management. This applies to the increasing number of children and adolescents in Europe with a chronic illness. For example, the incidence of childhood type 1 diabetes mellitus (T1DM) in Europe, now ranging from 3.9/100,000 cases per year in Macedonia to 57.4/100,000 in Finland [26], is rising rapidly. In the below 5-year-old age group, there is a doubling time of less than 20 years [13]. T1DM is associated with serious physical and psychological complications [8, 27], which may appear sooner or later, cause high morbidity and mortality, affect the quality of life, and increase health-care costs [14]. Complications can be prevented by performing self-management (e.g., monitoring blood glucose, recognizing symptoms and injecting insulin). However, self-management is not an easy goal to attain for young patients. First, it requires motivation and long-term perseverance, in order to become a way of life. However, children’s illness regularly causes feelings of embarrassment (approximately 25 % of the youth involved in a study of Peyrot [27]), and negative effects on school performance and psychological well-being. Improving the way they feel about diabetes, might be a first step in improving the self-management. Second, the children need not only to learn to self-manage their lifestyle-related diseases to improve their situated health-related habits, but also to be prepared for the physical and social changes at adolescence. Third, the specific self-management goals of children and adolescents are strongly affected by a diversity of personal and environmental factors, such as the childs developmental stage, parents support and health care providers. So, children and their social environment have to find a personalized strategy to establish pervasive self-management.
1.2 Improving Self-Management
There is a broad source of literature on theories that are relevant for self-management support: Changing behavior [9, 20, 30], persuasive design [11], gaming theory [7], education [40] and behavior change support systems [24]. These theories have some common principles. According to the first principle, intrinsic motivation is key and requires that someone feels in control of the situation (experience autonomy). This can be reached for instance by providing variation and influence of dialog. The second principle emphasizes the feeling of competence: The user should feel capable of reaching an objective. This principle originates from educational and gaming theory [7, 40], and from behavior change literature [9, 20, 30], stating that relevant activities and objectives should be provided, which are challenging and achievable, and for which positive feedback should be provided. The third and final principle concerns relatedness: Education and self-management are improved when there is a relation between tutor and trainee. The tutor can be a peer or teacher with whom a form of relatedness (or rapport) is build up [20, 25, 40]. The three factors: autonomy, competence and relatedness are the building blocks of the self-determination theory (SDT) [9].
Anzeige
1.3 Social Robots
Social robots show human-like (social) characteristics, e.g. they express emotions and use natural cues as gaze to share point of focus [12]. For prolonged self-management support, rapport should be build up between child and robot resulting in a positive effect on relatedness [4]. In Zhao et al. [25], several behaviors are identified to create rapport between an agent and a person. Examples are the initiation of mutual self-disclosure, praise and acknowledgement, and referring to shared experiences. It is interesting to note that these behaviors are also prescribed in behavior change methods, e.g., express empathy in Motivational Interviewing [21]. So, the social robot can be viewed as an embodiment of a behavior change support system [24]. Such robots are being used for behavior change support, for instance, to support persons with autism [1, 29], to acquire a healthy lifestyle [33], and to educate persons (e.g. [18, 35, 36]). A robot has a rich set of possibilities to incorporate behavior-change methods from social sciences, but the specific translation from these methods to a coherent and concise set of robot functions is complex and difficult to evaluate.
1.4 Situated Cognitive Engineering
The European ALIZ-e project aimed at a social robot that 7 to 11 year old children could use recurrently and possibly help these children to progress on self-management (i.e., autonomy, competence and relatedness, [2, 3]; see Sect. 1.2). An iterative situated Cognitive Engineering method was applied [22], to (i) derive use cases, requirements and claims for the self-management support (i.e. the design rationale), and (ii) build prototypes to test and refine the design rationale. The tests were conducted at schools and hospitals, focusing on specific parts of the design rationale, i.e. one or more “core functions” of the social robot that were hypothesized to have effect on relevant SDT-factors. For example, the idea that relatedness is stimulated by having a background story for the robot [39]. These functions were studied in different set-ups and with different groups of children (e.g. classmates at a school, or patients in a hospital). Often it was not (yet) required (for a first test and refinement cycle) to involve the target group, children with diabetes. This paper takes the lessons learned from these tests to design a general scenario, incorporating a variety of use cases. This way, an integrated set of core functions was prototyped and tested with children with diabetes in a hospital (i.e. the real target environment).
The next sections provide an overview of the earlier experiments conducted and their results. The current study incorporates the “proven” functions and makes use of the insights on the experimental setup that we built up in these experiments. The resulting social robot and scenario are evaluated with diabetic children in a hospital setting, studying the influence on autonomy, competence and relatedness. Furthermore, the perceptions and opinions of the children, their parents and their medical caregivers on the short and long-term are investigated. Conclusive evidence on the effects of the specific metrics could not be found, but the interactions with the children, parents and caregivers during the evaluation and afterwards gave valuable insights. Parents and caregivers became more enthusiastic over time and reported results in increased self-management and lower thresholds in hospital visits.
Fig. 1
The games: quiz and sorting game (a) The quiz (b) The sorting game
×
2 Lessons Learned from Previous Experiments
Over four years, several tests were conducted, in which children interacted with a social robot (Philips iCat or Aldebaran NAO) and performed one or several activities with the robot. These activities were designed to examine the effects of specific support functions, e.g. on specific learning objectives. Four educational activities were developed. The first was a Trivial Pursuit®based quiz in which robot and child played against each other. This educational quiz had a textual and competitive nature Fig. 1a. The second activity was an educational sorting game Fig. 1b, in which the child and robot classified objects in categories and could cooperate to reach the highest classification score. Due to its collaborative nature and visual orientation, the sorting game involved another learning style than the competitive and textual quiz (whereas, they could support the same learning objective). The third educational activity entailed different versions of movement games [31], which could address the same learning objective, but in a kinesthetic learning style. The fourth activity used educational videos that are both visual and aural. With this variety of activities, the social robot could support a variety of learning styles [10]. Next to these educational activities, there were “intervening” activities, such as small talk, to establish continuous child–robot interactions. All robot support functions were designed to address the objectives of SDT: autonomy, competence and relatedness. Table 1 provides an overview of the relevant experiments, their relations to the objectives of SDT, the context (setting and users) in which the experiment was conducted, the results and the transfer of these results into the integrated social robot (that will be tested subsequently).
Anzeige
According to SDT, a feeling of autonomy can be enhanced by providing choices. To stimulate this, the ALIZ-e project aimed at providing numerous activities that robot and child could do together. The quiz and sorting game were developed to support this. They both focus on education, but where the robot and child are playing against each other in the Trivial Pursuit®based quiz, in the sorting game they have to cooperate to get the highest score. In [15] it was shown that the possibility to switch between activities is beneficial for the motivation (see experiment 1 in Table 1).
The second factor of SDT, competence, can be supported by adapting the difficulty of the exercises to the child [16]. This adaptation proved to be beneficial for the motivation of the children (see experiment 2). It should be noted that the robot was not an expert in this interaction, i.e., the robot made the same amount of errors as the child [32]. Showing that the robot was not an expert was emphasized by making the robot exhibit thinking behavior [42]. Overall, this resulted in a positive experience of the robot (see experiment 3 and 4 in Table 1). In addition to competence, experiment 3 and 4 also addressed relatedness by encouraging self-confidence.
The third pillar of SDT is relatedness, meaning that the robot is experienced as a “pal”. Firstly we made sure that the robot can exhibit recognizable emotions [6, 19] (see experiments 5 and 6). We also looked at adapting the robot to the personality of the child [38], but we found that personality is probably not a good aspect to adapt to (experiment 7). We still expect that adapting to energy level, and perhaps modulating the energy level of the child will support the relatedness, but this was not evaluated. We did evaluate the adaption of robot’s emotional state to the state of the user and state of the situation (within boundaries) [37]. The results from this experiment showed that children who interacted with the robot that adapted its emotional state to the child and situation, showed more, and more positive, emotional expressions than children who interacted with a robot that did not adapt its emotions to the child and situation (experiment 8). However, recognizing child’s emotions in an interactive situation is still very hard. Therefore, we studied the effects of remembering small facts about their life (e.g. name, hobbies, information provided in a previous session) [5]. This is rather easy to implement and proved to have a very positive effect on the children (see experiment 9). Another easy to implement functionality is that the robot tells something about itself (e.g. age, hobbies), which proved to increase the willingness of the children to disclose information about themselves [39] (experiment 10). Finally, we looked at the willingness of children to touch the robot [34]; experiment 11 showed that they are quite willing.
In addition to the conclusive results, interesting observations were acquired during the experiments that are relevant for the further development of the robot. For example, changing activities by the robot and the child themselves proved to be stimulating (e.g., to transfer from quiz to sorting game without the help of the experiment leader [15]; see results experiment 1 in Table 1). Another observation was that providing a confined, shared environment for the robot and child proved to reduce child’s feeling of being observed and part of an experiment [34] (experiment 11).
Table 1
The 11 experiments that examined specific robot support functions for child’s self-management with their relations to the self-determination theory (SDT), the location of the experiment, the participants and the results
Nr.
Experiment
SDT objectives focus
Location
Users (nr, age)
Results (implemented in current evaluation y/n)
1
Multiple activities (quiz, sorting game) in hospital setting [15]
Autonomy
Hospital
Non-diabetic children (13, age 7–11 )—hospitalized
Multiple activities are beneficial (y) child and robot should go from one activity to another (y)
2
Difficulty of math assignments adapted to performance child [16]
Competence
School
Children (20, age 9–10)
Adapting the difficulty of exercises to the child increases motivation (n)
Robot is able to adapt its performance to performance child (n) fallibility is not proven effective, but theory is convincing (y)
4
Make the robot think about its answers. The robot takes some time and expresses thinking behavior [42]
Competence
School
Children (26, age 9–11)
The thinking robot is experienced as faster, more humanlike and more likeable, without decreasing perception of intelligence, trustworthiness and autonomy (y)
Emotions are recognizable and add to likeability (y)
6
Compare capability to express emotions between robots [6]
Relatedness
School
Children (14, age 8–9)
Emotions of NAO are equally well recognized as thos of iCat (y)
7
Adapt robot personality behavior to personality child [38]
Relatedness
School
Children (16, age 7–9)
Personality was very hard to determine (no correlation between what child, parent and teacher indicated) so not a good factor to adapt to. (n)
8
Adapt robot emotions to child emotions and state of activity (within boundaries) [37]
Relatedness
School
Children (18, age 8–10)
Children that interacted with the robot that adapted its emotional state to the child and situation showed more, and more positive, emotional expressions than children that interacted with a robot that did not adapt its emotions to the child and situation (y)
9
Make robot remember small facts about their life (e.g. name, hobbies, information provided in a previous session) [5]
Relatedness
Hospital and At home
Diabetic children (30, age 6–12) - not hospitalized
Very positive effect on relation children towards robot (y)
10
Make robot disclose information about itself (e.g. age, hobbies) to stimulate disclosure of child [39]
Relatedness
At home
Diabetic children (6, age 9–12)—not hospitalized
Increased willingness of children to disclose information about themselves (y)
11
Ask the child to touch the robot in an interactive move session [34]
Relatedness
School
Children (22, age 9–11)
Most children like to touch the robot (y) The children like an enclosed environment where they are more alone? with the robot (y)
3 Constructing an Integrated Set of Child–Robot Activities for Hospital Visits
Table 1 provides an overview of the 11 experiments that examined the specific robot support functions for child’s self-management with their relations to the self-determination theory (SDT), the location of the experiment, the participants and the results. The following subsections will elaborate on these results and describe how they will feed into the next version of the robot and the set of child–robot activities for returning hospital visits.
3.1 Child–Robot Interaction Environment
Based on the knowledge gathered in experiment 11 [34] we developed a physical setup for this evaluation. Firstly, we used the robot playground as used in [34] again. The playground (see Fig. 3) consists of three walls of 150cm high on which a robot landscape is depicted in soft grays. The floor consists of grey playtiles and one red and one blue depicting the positions the child should sit for the different games. All cables are hidden under the floor and behind the walls and two cameras are unobtrusively placed behind the walls so they just peek over it. The playground provides a shared environment for robot and child and since we did this experiment inside the hospital it also makes the surroundings more friendly. Furthermore, because children sit on the ground with the robot they are naturally on the same level as the robot, which is different when the robot stands on a table and the child sits at the table, which is also more static. Finally, the shared environment closes of the rest of the environment more, so the experimenter, who is in the same room, is easier to forget about.
3.2 Child–Robot Activities
Next to the environment we made sure the interactions were in concordance with what we learned from previous experiences. The evaluation was a wizard-of-oz evaluation, which meant that the experimenter/wizard did the speech and state recognition of the child and there was a protocol that was followed that described the possible dialog and behavior actions. The wizard had some freedom to put in new text for the robot to say. The wizard had camera images from the playground, could switch from camera dependent on the activity, and had an elaborate wizard interface to direct the interaction. Overall, the activities consisted of three educational child–robot activities (quiz, sorting game and video watching), two intervening child–robot interactions (small talk and walking), and specific tests to assess the children and their experiences.
3.2.1 Educational Activities
The child and robot could do three activities together, following experiment 1 that concluded that multiple activities are beneficial [15]. The two games as developed within the ALIZ-e project and an educational video. The quiz was based on Trivial Pursuit®. Child and robot each stand on opposite sides of a tablet in a kind of see-saw construction (see Fig. 1a). The tablet is turned towards the robot and it can then ask the first multiple choice (A–D) question. After posing the question the robot turns the tablet towards the child and the child can answer, by saying the answer out loud (no touch). The robot reacts on the answer and congratulates when it is correct and provides the argumentation when it is incorrect. There is no judgment when the answer is incorrect. Then the next question appears on the tablet and the child can pose it to the robot. The robot thinks about the answers it provides (experiment 4) [42] and makes errors (experiment 3) [32]. The game can be set up competitive, but we did not incorporate a scoring mechanism.
The sorting game shows pictures on a large touch screen (see Fig. 1b), the pictures need to be swiped into one of two categories that are named/depicted on the sides of the screen. The categories are for instance “high in” and “low in” carbohydrates and pictures shown on the screen are “a salad”, “chips”, “bread”, “sweets”, “milk” etc.. Child and robot stand on opposite sides of the table and they can both, one at the time, swipe a picture in the correct category. The aim here is to get a high score together, so it is a collaborative game setup. During the game the robot acknowledges the actions of the child with exclamations as “too bad”, “you did great”.
The difficulty of both the quiz and the sorting game was not adapted to the users’ performance although it was found to be effective (experiment 2) [16]. We did not do this because of a limited number of questions/assignments per session and a high variability between children. The questions/assignments were related to diabetes and thus relevant for the children.
The final activity is not a game, but an educational video the robot and child can watch together. The video is for instance about the symptoms of high blood glucose levels (a “hyper”).
After a certain number of questions of the quiz (8), or a certain amount of time with the sorting game (5 min) the robot initiated a change activity dialog. The child could then choose to proceed or change activity, although in the first and second session they had to do all activities so there was a time limit on how long they could do each game (10 min max). The child could also initiate the dialog to change the activity. When this was really soon after starting the activity, the robot tried to convince the child to do it a little longer (“just a few more questions”), otherwise it would agree on changing.
3.2.2 Small Talk
Based on experiment 9 and 10 [5, 39] we incorporated small talk in the evaluation. At the start of the evaluation the robot asked the child some personal information: Name, age, hobby. The robot did also ask if the child had questions for the robot, so it could also answer questions about its age and hobbies. Furthermore, the robot asked at the end of the first and second session if the child had plans for the coming weeks (until the next session) and referred back to these in the next session. Finally, during the activities the robot asked questions about diabetes. The robot for instance said “The holiday period seems to be really hard to me, with all the candy and strange food, how do you deal with that?”. During the small talk and the activities the robot displayed emotions that correlated with the situation (experiment 5, 6 and 8 [6, 19, 37]).
3.2.3 Walking
Because we did not want a detrimental effect on the interaction when switching activities, because of interference of the experimenter (experiment 1 [15]), we decided the child was responsible for getting the robot from one activity to another (see Fig. 2). We thought this would work because experiment 11 [34] showed no hesitation of most children to touch the robot. We explained how to walk with the robot, but when some children started to lift the robot we also accepted this. Something else that came up after a few of the first sessions were finished, was that the robot fell over sometimes and most children felt the need to help it up. Therefore we added a function that made sure the robot would not hurt the child, shutting down the automatic stand up function and removing motor stiffness, so that the child could support the robot standing up. We also explained to the children how they could help the robot in standing up by putting it in sitting position.
Fig. 2
Walking with the robot
Fig. 3
The robot playground
×
×
4 Evaluation
In order to get a feeling of how diabetic children interacted with the NAO when different activities are offered and physical interaction is possible we carried out an experiment.
4.1 Evaluation Method
4.1.1 Participants
17 diabetic children in the age of 6–10 (M\(\,=\,\)8.24, SD\(\,=\,\)1.25) participated in the experiment. They were selected by their diabetic nurses of the Meander Medical Centre (Amersfoort, the Netherlands) and on basis of the parents willing to come three times extra to the hospital. All children got the diabetes diagnosis more than a year and a half ago, the range was 23–108 months (M\(\,=\,\)51, SD\(\,=\,\)29,64). Most children used a pump to regulate their insulin intake (11), the others used insulin injections (6).
4.1.2 Materials
To execute the experiment in an adequate way the following materials are needed for the experimental setting: The child with the robot on the robot playground and the execution of the experiment including measurement material.
Robot playground: playtile floor of 2 \(\times \) 3 m\(^2\) with walls (Fig. 3)
See-saw tablet holder, a device enabling turntaking by flipping the tablet
Samsung Tablet
15\(^{\prime \prime }\) screen to watch little movies about diabetes
27\(^{\prime \prime }\) television touch screen with table legs, to play the sorting game
Questionnaires
Wizard Laptop
Movie Laptop
An extra screen to watch interaction
Cameras to record interaction
3 NAO robots (2 minimum needed for third session and backup when technical failures occur)
4.1.3 Measures
We used observations, tests and questionnaires to quantify and qualify the interaction with the robot. Tests
Knowledge test This questionnaire is used to assess whether there is knowledge improvement. This test is filled out before the first and after the last interaction and consists of 32 knowledge questions (e.g. What is important for you to know about your physical education class? (a) If you’re going to do something fun, (b) If it is active or calm what you’re going to do, (c) If you are going to play football, (d) If you’re clothes look good: b is correct). The questions one until eight occur in the first session of interaction on the tablet, questions nine until 16 in the second session and questions 17 until 24 in the third session (for the children who chose the quiz). When questions or answers were not understood or the children were not able to read they received help.
Self-efficacy test The SE card-sorting questionnaire is used to assess the current autonomy of the child. To measure SE, a card sorting questionnaire based on Karoly and Bay [17] is used together with diabetes-care activities proposed by the diabetes specialists of the Meander Medical Center.
Memory test With the aid of a memorizing task we examine whether children memorize more information given by a familiar robot, as is expected when intrinsic motivation is higher due to a peer teacher that applies SDT strategies [23]. In the third session every child listens to two robot stories. One story is based on the English Wechsler Intelligence Scale for adults (Williams [41]) and the other one thought up using the same build up. One story is given from the familiar robot (called Charlie) and the other story is provided by another NAO robot (called Robin), who is introduced as a friend of Charlie. This robot is exactly the same as Charlie, but has a different voice and wears a grey striped shirt. The order of the stories and the robots is counterbalanced. After each story there is a short recall memory test. First the children are asked to reiterate the story as best as they can (immediate free recall). After this they are asked nine questions about the story (Immediate cued recall). An example of such a question is: “what was the name of the lady in the story?”
Questionnaires
Fun Questionnaire To measure the pleasure and fun the children experienced the children filled in a Likert scale questionnaire about the robot and the activities. First there were three 7-point questions on fun with the robot, quiz and sorting game, after which four 4-point questions were asked related to different aspects of the robot. The questionnaires used were based on the Smileyometer from the Fun Toolkit of Read and MacFarlane [28].
SDT Questionnaire To measure the feeling of self-determination we asked the children 10 questions on a 7-point Likert scale. Question 2,3,8 and 9 were regarding feeling of competence, question 4,6,7,10 were about feeling of relatedness and question 1 and 5 were related to feeling of autonomy.
Observations
Game preference In the second session the children could say which game they preferred and were asked if they wanted to start with this game and in the third session they could only choose one game.
Online analysis and offline video and logging analysis For the analysis of the whole interaction in each session we used notes that were taken during the interaction, video analysis and analysis of the logs. We looked at walking, time with activities, game order, attention of child, interaction with robot (talking general, talking diabetic related, touching), reaction on technical failures, empathy with robot, and how much the experiment leader is involved.
4.1.4 Procedure
Every child had three sessions of about an hour in the hospital. These appointments were at least 14 days apart (see Fig. 4).
Fig. 4
Planning for the three sessions
×
In the first session the NAO robot, called Charlie, is introduced as a robot that helps children to manage their diabetes but still has to learn many things about diabetes himself. The experiment leader explains the activities in short and shows how the children can walk with Charlie. The interaction with Charlie starts with small talk and walking followed by one of the games. With the quiz Charlie has to be put exactly in front of the bars on the ground to be able to turn the tablet. In each session at least eight questions are played so that after three sessions they practiced 24 of the 32 knowledge test questions (if they chose to do the quiz in the last session). The sorting game is on the other side of the playground on a large touchscreen. Several pictures are shown and the child and NAO have to put them in the correct category (on one of the sides of the display). Examples of categories are: hyper/hypo, low/high carbohydrates. During each game open questions related to diabetes are asked to support self-disclosure (e.g. “Did it ever happen to you that you had a hypo or hyper and did not notice? How come did this happen?”). In between the games in the first and second session the children can watch an 1-min movie about dealing with typical diabetes situations, which is presented on a 15” screen. Dependent on the time left another short movie can be presented to the child after the games. After the interaction with the robot the children always fill in a questionnaire concerning judgment of the robot and the games they have played. The first session starts with the quiz. In the second session the children are allowed to choose with which game they want to begin with but they have to play them both. In the third session only one game is played, chosen by the child, because of a new scenario where the children meet Charlie’s friend Robin. Both Robin and Charlie tell a short story after which the children have to do a test with free and cued recall about the story.
5 Results
Below we will describe the results from the evaluation. These results are divided in results that can be directly derived from the instruments and observations used in the evaluation and in feedback we got afterwards.
The tests were analyzed using t-tests and the questionnaires using the non-parametric Wilcoxon and Friedman tests. Game preference was counted and compared between the second and third session. The video and logging analysis was performed using Grounded Theory as starting point. This was because the 17 children differed in age, phase in their illness and interaction with the robot so much that we couldn’t compare between them. What we could do was analyzing the data looking for similarities and differences, to create preliminary user profiles, on which the robot could adapt its interaction in the future. All videos and logging files were watched and we looked at similar behavior between the participants on aspects as speech and touch interaction (time spent, manner of interaction, extravert behaviour etc.).
5.1 Tests
The self-efficacy test is excluded because most children had some difficulty filling it in. Furthermore, the test took too long to do a pre- and post test.
Knowledge test Questions 7, 8 en 18 are excluded because we noticed that multiple answers were correct. A paired sample t-test shows that there was a significant difference in knowledge acquisition between the pre- and post test for the questions that were presented during the experiment (1-24). First session M\(=\)11.35, SE\(=\).77, second session M\(=\)13.7, SE\(=\).66 and a paired t test t(16)\(=\)5.6, p<0.001 (2-tailed). The final eight questions (25-32) did not show significant improvement t(16) \(=\)1.19, p\(=\).25 with M\(=\)5.94 and SE\(=\).34 for the first session and M\(=\)6.29 and SE\(=\).44 for the second session.
Memory test We did an independent samples t-test to test whether there is a significant effect of the robots in the immediate free recall and in the immediate cued recall (see Fig. 5). There are no significant differences assessed between the scores reached after the stories told by Charlie and the scores reached after the story told by Robin in the immediate free recall (\(p=.114\), \(p=.521\)) and in the immediate cued recall (\(p=.869\), \(p=.306\)).
Fig. 5
Story recall comparison between Charlie and Robin
×
5.2 Questionnaires
Fun We had separate questions on fun with robot, quiz and sorting game. Over the sessions these did not change significantly. The same was true for the separate questions on interaction with the robot (see Table 2).
Table 2
Fun questionnaire means and (SD) [* 1 NA due to technical failure sorting game, ** for quiz: 2 NA and 6 children who filled in something while they didn’t play the quiz, for sorting game: 3 NA and 6 children who filled something in while they didn’t play the sorting game, *** 1 NA (missed question)]
Question (scale)
Session 1
Session 2
Session 3
How much fun did you find Charlie the robot? (1–7)
6.5 (0.87)
6.8 (0.43)
6.8 (0.44)
How much fun did you find the quiz? (1–7)
5.8 (0.88)
6.2 (0.66)
5.7 (1.16)**
How much fun did you find the sorting game (1–7)
5.9 (1.14)
6.1 (1.12)*
5.7 (0.73)**
How friendly did you find Charlie the robot (1–4)
3.9 (0.33)
3.8 (0.39)
6.1 (0.40)***
How well could you play together with Charlie the robot? (1–4)
3.6 (0.51)
3.8 (0.39)
3.8 (0.47)
How “cosy” is Charlie? (1–4)
3.7 (0.44)
3.9 (0.33)
3.8 (0.62)
How warm (hospitable) is Charlie? (1–4)
3.5 (0.62)
3.4 (0.61)
3.8 (1.35)
Self-determination For the self-determination questionnaire we aggregated the questions related to competence, to relatedness and to autonomy per session.
Competence Overall, 49 % of the children rated their feeling of competence a 7 (highest) and only 4 % rated their competence under 4. In session 2 this was 56 and 7 % and in the third session 50 and 4 %. This means that no improvement was possible for almost half of the children and only very little for the children who scored initially under 7.
Relatedness We performed the same procedure as for competence and counted the number of times a 7 (highest) was chosen. 69 % of the time children felt very related to the robot and only 6 % chose a rating under 4. In the second session this was 76 and 1 % and the third session 74 and 3 %. So as with competence the ratings were already so high in the first session there was little room for improvement, 54 % of the questions were rated a 7 on all three sessions.
Autonomy The autonomy was rated a 7 (highest) for 38 % of the time in the first session (15 % under 4), 44 % in the second session (6 % under 4) and 53 % in the third session (6 % under 4). Because of this increase we performed a Friedman test, but this was not significant p = 0.29 (df = 2, chi2 = 2.45).
5.3 Observations
Game Preference In the second session 9 of the 17 children chose the sorting game as their favorite and 8 chose quiz and they also agreed starting with this game. In the third session 8 children chose to play the sorting game and 9 the quiz. 16 of the 17 children chose the game they preferred in the second session to play in the third. Only one child switched from sorting game to quiz.
Video and logging analysis From the video and logging analysis we extracted five user profiles as shown in the following and we did some additional observations.
User profiles The profiles were based on observations made during the experiment itself and observations from the videos afterwards. During the experiment the wizard, who was the same in almost all sessions, made notes about the behavior of the child in the experiment. Afterwards the same person identified some aspects, based on the notes and rewatching a few videos on which the children could be categorized in profiles. The scoring aspects were discussed with colleagues. Then taking these aspects all sessions were watched and scored. Aspects we looked at were related to dialog and actions of the robot, e.g. naming the child, falling. But also to dialog and actions of the child, e.g. reaction on falling of the robot, attention towards robot, time spent in activities, talking with robot (only telling or also listening), walking with robot, reaction on diabetes related questions. Finally we did also some general observations about the child, e.g. happy, open, shy, technology minded.
1.
Children who “just deal with it”(pp3, pp10, pp12, pp13, pp16) In this group there are children who know very much about diabetes and how to deal with it. They can tell about it in an open manner, even about the difficult parts (see Table 3). They seem to feel good and do many things on their own. In the group of children who “just deal with it” there are also children whose parents have diabetes. The children who indicated that their parents have diabetes seem to be much more relaxed and open for diabetes related questions and providing information to Charlie in a positive way. Diabetes for these children seems to be a shared (and not problematic) lifestyle together with a parent.
2.
Children who feel to fall outside the group(pp2, pp9, pp11) Children who seem to feel not that comfortable yet with having diabetes and the integration of it in their life belong to this group. Different reasons can be listed for this feeling. For example when children do not know enough about their diabetes, cannot connect the consequences of the diabetes to their feelings and are therefore more dependent on their parents. In the interaction this becomes clear by difficulty answering the open questions related to diabetes. They also see Charlie immediately as a friend, this is shown by having a picture of Charlie above the bed at home (pp2), having lots of empathy for Charlie when it falls (pp9) and more then passing interest in how many friends Charlie has (pp11).
3.
Children who are afraid to make errors(pp4, pp5, pp14, pp17) When children look away very often during interaction, give answers which are not consistent with their behavior or are ashamed to say anything, it seems that children react only like that because of someone listening or watching (for example Table 4). These children seem not that sure in what they know about diabetes and do not dare to say something, because it could be wrong.
4.
Children who are shy(pp7, pp8, pp16) These children take a longer time to tell something or do something with the robot. Often they whisper their answers, or just laugh a bit uncomfortable.
5.
Children who have difficulty with multitasking(pp1, pp5, pp6, pp17) Some children in this experiment were still very young and had difficulties with talking with Charlie and playing the games at the same time. Sometimes these children could not read the quiz questions themselves. The experiment leader plays a big role in these interactions. Social desirable behavior is almost unpreventable in those situations. In the most cases they also know less about diabetes than the other children and do less diabetes related actions on their own.
Table 3
Example of ‘children who “just deal with it”’
Kind (pp12)
Ik ga heel goed opletten, wat ik eet en ik kijk goed op de verpakkingen. En dan onthou ik dan. Als ik bijvoorbeeld bij Sinterklaas pepernoten wil eten, weet ik hoeveel in 50 g zit en dan hou ik dat in mijn hoofd als ik dan de volgende keer 100 g wil eten weet ik dat dubbele moet doen
Charlie
Oh, wat goed zeg! Nu ik dat weet, kan ik het ook aan andere kinderen leren
Kind (glimlacht)
Oh dat is fijn!
Translated
Child
“I’m very careful with what I eat, look on the packaging and remember that. So If I want to eat ginger nuts at Sinterklaas for example I know how much sugar there is in 50 g. I keep that in mind and when I want to eat 100 g I know that I have to do twice as much insulin”
Charlie
“Oh, great! Since I know that now, I can tell it to other kids”
Child (smiling)
“Oh, that is good”
Table 4
Example of ‘children who are afraid to make errors’
Kind (lijkt arrogant/onzeker en kijkt vaak weg) (pp4)
Nou, als ik iets wil eten, dan spuit ik gewoon
Translated
Child (seems arrogant/unsure, often looking away)
So, when I want to eat something, I just inject insulin
Other observations In general some children touch the robot from the first meeting on, curious about how it feels. Especially in the last session Charlie gets many questions of how it works. All children are interested in unpredictable facts about Charlie as for example the name and colors of Charlie’s soccer club and the outcome of the last game. Furthermore, compliments seem to support all children: They react positively on them, some react more reserved whereas others give the robot compliments in return immediately.
Walking the robot is not very easy, at least not to bump into anything, but it is appreciated by most and when it goes not fast enough they just carry it to the intended spot. Also the falling seemed to support most children in feeling useful, but not all children liked to help the robot after it fell. All children had to help the robot to the other activities and all children experienced at least one fall during their three sessions. For some children this occurred more often than for others. Our feeling was that although helping to stand up was beneficial the falling had a negative influence when it occurred often.
In the dialogs we saw some progression in what was disclosed towards the robot, they really wanted to tell the robot about their experiences in between the visits. Very noteworthy is that 4 children gave a present to the robot (drawing, paper craft, loom bracelet and World Football Cup goodie) (see for example Fig. 6).
Fig. 6
Drawing and paper craft gifts
×
5.4 Feedback After Evaluation
At the moment of completing this paper the experiment has finished a year and a half ago; since that time we received great feedback from parents and medical staff. Parents have told us of more independence since the three 20-min interactions session. Medical staff tells us that children still ask when the robot returns and that they notice children are more at ease at the hospital since the experiment. In follow up contacts we noticed that parents, children and medical staff are more willing to participate in a follow up study than they were to participate in this study. This is also apparent in the fact that the Meander Medical Centre is now part of the H2020 project PAL that also looks at the use of the robot, in physical and virtual form, for children with diabetes.
6 Conclusions and Discussion of the Evaluation Results
6.1 Tests
After negative experiences with other questionnaires, we decided to use this self-efficacy questionnaire with the sorting cards. This method seems to work well: It encourages the children to think about their answers and vary them. But the questionnaire was not enough adapted to the target age and took too long to fill in. So although it did not have the desired result now, we would like to refine it and use it as pre- and posttest for self-efficacy in the future. In the Netherlands there is a list of “Know and Do” objectives for different age groups (6/7, 8/9, etc.); we are looking in to using this to measure the level of self-management. Of course we will also look for alternatives to measure variation in self-efficacy related to diabetes over time. It should be noted that parents and medical personnel indicated (after the experiment) that self-efficacy was improved. One of the parents for instance told that their daughter made more decisions on her own, like adapting the insulin before a meal because she wouldn’t eat a lot of it. The parent said that the fact the robot made errors did have a positive effect. Furthermore, although not significant, there was an increase in autonomy according to the questionnaire.
The knowledge test had good results, but improvements are possible. Some (more interesting) questions had multiple possible answers, because in many situations there are multiple solutions for the problem at hand for diabetics (this is just one of the things we want to learn the children). Also the reaction to high or low bloodsugar is dependent on the situation: Illness, stress, physical activity and food influence the bloodsugar and to keep the variation at a minimum it is necessary to know why the body reacted in this way to come to the best reaction. Furthermore, we noticed that children answered lifestyle questions truthfully. So when asked how they handled a situation like telling a parent of a friend they had diabetes, they did not provide the “correct” answer, which was very obvious (“I do x because then I show I’m the boss of diabetes”), but said they rather not tell because it would make them different. We were very suprised, but also happy with this. We rather have the answer about how they handle such a situation so that we can make them understand why they should change behaviour than that they provide the “correct” answer.
The memory test did not result in a significant difference between the familiar (Charlie) and unfamilair robot (Robin), but we did see some opportunities to improve the test. First we need to make absolutely sure that both robots are equally understandable, while speaking with different voices and we should use a validated, for the specific age group, verbal memory test.
6.2 Questionnaires
All questionnaires suffered from the same problem, a ceiling effect. A score below the 6 was low which makes it impossible to have an increase over time. Next to this we saw that the sorting of cards in the self-efficacy test had a positive effect on thinking about a question, whereas some items of the questionnaires stimulated putting crosses automatically. This could be seen for instance in the questions of session 3 where many of the children (12 out of 17) answered questions about the activity they did not perform. It keeps being a challenge to have questionnaires that are informative, but they are still an important measurement method, so we will keep adapting them and hope to create an informative questionnaire. Furthermore, we will look further into ways to decreasing the effect, like make the answering more tangible (e.g. no cross but moving something to the answer), more forced choice, implicit association tests, providing parents with questionnaires for some effects, longer evaluation periods, and more.
6.3 Observations
6.3.1 Game Preference
It was nice to see that some children preferred the quiz while others preferred the sorting game. This encourages us to proceed with having different activities that are performed with the same robot to reach the same objective and that which activity is performed depends on the child’s preference, state and current objective.
6.3.2 Video and Logging Analysis
User profiles The user profiles indicated in this experiment are a starting point for us to focus on some parts of the interaction and see if we can recognize these same profiles in another experiment or that they need to be adapted. The profiles as they are now, are solely based on the interpretations of one coder and thus need to be verified. After a set of stable user profiles is identified we want to use these profiles in the future to make a fast adaptation to the user possible. Below we provide per user profile a first idea on how the user profile influences the adaptation.
1.
Children who “just deal with it”(pp3, pp10, pp12, pp13, pp16) The robot can tell the children who are more uncomfortable with their diabetes how these children could deal with it. The children mention that the robot needs to know more and get a teacher role which can give them more self-confidence. This group is challenging for the interaction because in particular the children who are easily comfortable in the interaction with the robot are also the first who get bored by the robot and its games. Fortunately, this group seems to be interested in a robot and how it works. In the interaction with this group this could be taken advantage of. Although the children in this group are already quite confident with their diabetes they might benefit from short interactions to provide them with a bit more confidence to take the next step in self-management. This idea is fed by the feedback we got from some parents with children in this group.
2.
Children who feel to fall outside the group(pp2, pp9, pp11): For these children the robot has to be a real friend. Remembering what the children said adds great value. It seems to be nice especially for these children to share interests with the robot, for example playing cards (pp2) or wearing bracelets (pp11). The robot should combine friendship and dealing with diabetes. To not break the bonding with the child, the robot has to be careful with its questions and for example not ask a question like “What do you do with Santa-Claus, so many weird food, how do you deal with it?” in the beginning of the interaction to not bring the child in an unpleasant situation.
3.
Children who are afraid to make errors(pp4, pp5, pp14, pp17) The robot can show the children that it doesn’t matter to make errors by making errors itself. It can give the children self-confidence through playing the games and praise when the children did something good. The bonding can grow and the child can grow too.
4.
Children who are shy(pp7, pp8, pp16) When children are very shy, the robot should be patient, and should play and walk with the children instead of talking too much. Some children need more time to talk about difficult issues. The robot has to try to estimate such children’s state and help them managing their diabetes without being too pushy.
5.
Children who have difficulty with multitasking(pp1, pp5, pp6, pp17) To improve self-efficacy and knowledge with children who have difficulty with multitasking, the robot should catch the attention and hold the attention of the children. That is very challenging especially because children are very good in ignoring other things when they are engrossed in something else. The bonding with the robot could grow in first instance via playing and later via dialogue.
6.4 Feedback After Evaluation
The feedback after evaluation provided us with lots of information, but in a semi-structured manner. Our experiences during this experiment with small talk with parents and health care professionals when they were watching the sessions and afterwards has shown us the importance of involving them in a more structured manner. In the future we will do this by involving them more in the design and evaluation via focus groups, structured interviews, participation in the experiment and questionnaires.
7 General Conclusion and Discussion
7.1 Main Outcomes
Overall, the general scenario for educational and enjoying child–robot activities during returning hospital visits, proved to capture the lessons learned well. The children had very positive experiences in the three sessions of almost one hour (i.e., quiz, sorting game and video watching, and small talk and walking). The children, but also their parents and formal caregivers, showed positive experiences. Children enjoyed the variety of activities, built a relationship with the robot and had a small knowledge gain. Parents and hospital staff pointed out that the robot had positive effects on child’s mood and openness, which may be helpful for self-management. Based on the evaluation results, we derived five user profiles for further personalization of the robot, and general requirements for mediating the support of parents and caregivers.
More specifically, personalization to developmental age, interests and objectives of a specific child, proves to be important for both the interaction as the questions asked. Furthermore, we should not only focus on improving self-efficacy of the child, but also on improving confidence of the parents in their child. Many of the parents were over-protective. Involvement of children, parents and medical staff is thus essential. Fortunately we have seen that formal and informal caregivers changed from skeptic to enthusiastic, based on the reactions of the children who showed increased self-management and more positive hospital experience. The robot showed to have a new role for self-management that is different from that of the caregiver and peer. If the long-term effects follow the same line is to be seen, the positive attention the children received now in relation to their illness can already explain many of the beneficial effects of the robot intervention. On the other hand, if we can have such an effect with three 20-min sessions with a robot it is worth the effort.
7.2 Importance of Evaluation “in the Wild”
Performing an evaluation with children with diabetes in a care environment provided us with knowledge and experiences we could not have acquired doing evaluations at schools. We noticed that diabetic children’s experiences with the robot differed from “healthy” children. They seemed to be more open for social interaction with the robot and also the fact that the robot was not all-knowing and dependent on the child seemed to influence these children more than healthy children. This was the first evaluation the robot received gifts from children, which shows that there is some kind of bond/relationship forming. The shared space of child and robot added to this experience as did the dependence of the robot on the child when it fell or had to go to another activity.
Because the children were brought to the experiment by their parents who often waited in the same room as the experiment leader (outside the experiment room) it was the first time we could interact with parents for a longer period. We of course knew that parents of children in this age group are of a huge influence on the child, and that this might be even more so with chronically ill children, seeing it first hand does change how you look at this influence. There were parents who already said at the beginning that they did not know if their child could perform well in the evaluation and we saw this back in the shyness of the mentioned child that changed a lot during the three sessions. Furthermore, having a child with diabetes has tremendous influence on family life. So caretakers and social environment influence the child, but the child also influences his or her environment. In future research we will take the influence and experiences of family and social environment into account.
The evaluation took place in the room next to the coffee corner of the hospital staff involved in the care of the diabetic children. This was great because they could look through a window and see what was happening, but also talk to parents and experiment leaders while getting coffee and thereby getting a better feel of the aim of the robot. They could see the enjoyment of the children, and also see and hear that the robot will not substitute them.
One of the main challenges we found is that because of the bond the children seemed to form and the things they discussed with the robot it did not feel ethically right to strictly follow protocol. For example when a child discussed his or her problems with diabetes because of a birthday party the robot did not react with “I don’t understand”, but the wizard typed in a relevant comment for the robot to say. Due to this and technical problems, no session was the same and the applicability of inferential statistics was limited.
7.3 Future Work
This evaluation showed that parents, medical staff and children enjoyed working with the robot and saw advantages of the use. The next step is now to develop a prototype that can stand alone, might also be used at home (in virtual form) because there are only a few hospital visits, and that involves all stakeholders. This means we need at least a solution to deal with speech recognition and dialog management, personalization on at least child interests, developmental age and objectives towards self-management, and evaluating effectiveness so that care institutions can argue for the costs of using the robot. Currently, these aspects are being addressed in the European H2020 project PAL (www.pal4u.eu).
Acknowledgments
First we would like to thank the children and their parents, who had to come three times outside of regular appointments. Without the Meander Medical Centre and the support of the diabetic team there, especially Roos Nuboer and Mirjam Schouten who made the whole team enthousiastic and made sure that our schedule was full and tight, this experiment would not have been possible. Finally, not only the authors of this paper contributed to this experiment, but also Joris Janssen was indispensable as experiment leader and Bert Bierman for his never ending and never failing technical support. This work is (partially) funded by the EU FP7 ALIZ-E project (Grant Number 248116) and the H2020 PAL project (Grant Number 643783).
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.
