Introduction

Cyberspace was envisioned to bring “an age of affordable beauty” [1]^p3 where transformations to the self would be effortless. Within this space, people would be able to freely change their appearance and behavior in ways that best fit their goals. Researchers hypothesized that individuals would be able to construct and reconstruct their identities through computer-mediated communication [2]. Indeed, a brief look at the current online sociosphere appears to confirm these early prophesies. People strategically manage their posts on social media to promote desirable aspects of themselves [3]. Individuals even misrepresent certain physical aspects of their actual selves, such as height or weight, on online dating websites to present themselves in a more positive light [4]. According to Goffman, this desire to manage and control how other people perceive and subsequently treat the self is thought to be a part of everyday interactions [5]. In ‘performing the self’, individuals aim “to convey an impression to others which it is in his interests to convey” [5]^p4.

Walther’s hyperpersonal model of communication suggests that people are often able to present themselves in a more positive light via computer-mediated communication (CMC) than during a face-to-face (FtF) conversation, because they have the time to strategically select the information that they wish to disclose about themselves [6]. The model further proposes that these effects are particularly stronger for asynchronous CMC, as this form of communication grants people more time to monitor and edit the messages that they wish to send to the other person. Underscoring this argument, Toma, Hancock, and Ellison posit that online daters are able to ‘reallocate’ cognitive resources into creating a desirable dating profile in lieu of trying to manage their verbal and nonverbal behavior in a real-time face-to-face conversation [7].

Indeed, one of the reasons CMC was thought to yield hyperpersonal relationships was due to the relative absence of nonverbal leakages that may lead to undesirable interpersonal consequences [7]. While nonverbal cues are not necessarily linked with the communicator’s actual feelings or truthfulness (see [8] for review), people tend to rely on these nonverbal behaviors to detect deception and/or form interpersonal judgments [9–11]. Despite the significance of nonverbal cues in interpersonal interactions, controlling nonverbal behavior during a real-time conversation has been difficult to do in both offline and online contexts.

Recent advancements in virtual reality technology, however, have led to the possibility of strategically controlling and altering one’s nonverbal behavior in computer-mediated contexts (i.e., transformed social interaction) [12]. Multiple studies have explored the interpersonal outcomes of transforming nonverbal cues of virtual humans [13–16]. For example, [14] found that people who interacted with an avatar that had been programmed to engage in a long mutual gaze (4 seconds) with them felt more positively toward their partner than when the mutual gaze was short (2 seconds). However, with a few notable exceptions [14], [17–18], most of these studies explored the influence of computer-controlled agents’ nonverbal behavior and not that of human-controlled avatars. Furthermore, even studies that did examine the implications of avatar’s nonverbal behavior often used computerized agents that were introduced to participants as avatars (e.g., [19]). While this measure was necessary in order to maximize experimental control, it also imposed restrictions in terms of ecological validity, as the conversational content between the ‘avatar’ and participant was fairly limited.

Avatars are visual representations of their users in virtual environments and can be separated from agents by the source of control; avatars are controlled by actual people, while agents are controlled by computerized algorithms [20]. Due to this difference in agency, (perceived) avatars and agents often yield different levels of social influence. For example, Hoyt and colleagues [21] conducted a study wherein participants were asked to perform a learned or novel task in front of virtual humans. Despite the fact that these virtual humans were not representations of actual people, they found that participants’ performance of the novel task was impaired when they were led to believe that the virtual audience consisted of avatars, but not when they were framed as agents. In their recent meta-analysis, [22] also concluded that avatars exert stronger levels of social influence compared to agents. As such, the current literature on virtual humans’ nonverbal cues would greatly benefit from studies that incorporate avatars that are controlled by actual people.

Among other nonverbal signals, a “substantial body of evidence…indicates that facial expressions serve as rich nonverbal cues that can powerfully communicate important interpersonal intentions and motives” [23]^p730. Out of these facial expressions, smiling commonly indicates high levels of social interest and acceptance [23] and plays a special role in interpersonal communication [24]. Studies show that smiling is used as a way to form bonds with others [25] and that smiling behavior is more pronounced when people are engaged in social contact [26].

Perhaps because smiling signals positive interpersonal intensions, research consistently shows positive effects of smiling on communication outcomes; people feel happier and smile more in the presence of people whose facial expressions indicate happiness (i.e., smiling); in addition, smiling people are often given more positive interpersonal ratings than people with neutral expressions [27–28]. Krumhuber and colleagues [29] found that even people who exhibit a smile that is not perceived as genuine are rated more likeable, attractive, trustworthy, and cooperative compared to people who assume a neutral expression. However, targets with genuine smiles still received the most positive ratings.

Despite the significant interpersonal function of smiling, most of the research on avatar nonverbal behavior has been limited to the movement of the neck (and resulting head position and rotation) and eyes. Although some studies explored the influence of a smiling avatar or agent on interpersonal perceptions (e.g., [24], [30–32]), results were mixed; technological limitations may have hampered the virtual human’s smiling behavior, leading to low behavioral realism (e.g., static smile, intermittent transitions from neutral to smiling expression). In one study, Cafaro and colleague [30] had participants walk up to a smiling or non-smiling virtual agent in an immersive virtual environment, and found that participants judged the agent more positively when it was smiling. In contrast, Krämer et al. [24] found that while interacting with a virtual agent that smiled frequently led to an increase in smiling behavior, it did not improve ratings of the virtual agent. Verhagen and colleagues [31] used smiling and non-smiling static images to represent a virtual customer agent, and did not find any differences in evaluations based on smiling behavior. In addition to the inconclusive results of the aforementioned studies, it is also important to note that all of these studies were conducted using computerized agents, and not avatars controlled by actual people.

One study that did compare the effects of smiling behavior between agents and avatars found that while smiling agents received more positive evaluations than non-smiling agents, smiling behavior ironically led to more negative evaluations when participants were told that they were communicating with a real person [32]. These results are surprising considering the wealth of literature that point to the positive outcomes of smiles. While it is likely that there is more than one explanation for this unexpected finding, the authors suggest that one reason for these results was the limited technology at the time [32], wherein the experimenter activated a pre-programmed smile by pressing a button. The level of realism provided by such a smile may not have been sufficient for the relatively higher thresholds of behavioral realism people have for actual humans compared to computerized agents.

With the rapid advancement of facial tracking technology in the past few years, it is now less challenging—both computationally and financially—to render realistic facial expressions on an avatar. A number of new technologies allow virtual environments to use facial tracking data from webcams or specialized depth cameras to render the user’s actual expressions on a virtual character in real-time (e.g., Face Analysis, FaceRig, Faceshift, Faceware, Nito, and Pocket Avatars). More importantly, the interactants in virtual environments can now subtly alter and enhance these expressions. For instance, considering the positive social implications of smiling, the user can choose to enhance his or her avatar’s smile. This approach differs from that of previous smiling research in that it uses the interactant’s actual behavior as a baseline, instead of using a pre-programmed smile that is activated on a button press. If the user doesn’t smile at all, the avatar would not smile either; it is only when the user actually smiles that the avatar exhibits (enhanced) smiling behavior. Such an approach is more likely to allow for a more realistic transition between facial expressions as well as a higher consistency between verbal and nonverbal behavior. As such, the current advancement in technology offers researchers the possibility to study the implications of nonverbal cues in both a more controlled and practically relevant manner.

The Present Study

The primary aim of the study is to determine if ‘enhanced’ (more intense) smiles would yield more positive communication outcomes compared to ‘normal’ (consistent with participants’ actual) smiles. More specifically, we were interested in if enhancing the smiles of the interactants’ avatars would lead participants to feel greater positive affect and like their partner more. In addition, we were curious how enhancing the avatar’s smile would influence participants’ perception of their communication experience, namely social presence. Social presence refers to “the sense of being with another” [33]^p456 and is used to assess the extent to which communicators are aware of the presence of another person and have access to the other person’s affective and cognitive states [34]. Higher social presence is generally associated with higher satisfaction with the communication medium. Finally, we explored if participants in the enhanced smile condition would choose to spend more time on the avatar-networking platform (vs. FtF communication) when given the option to divide their time between different modalities. We hypothesized that communicating via avatars that ‘enhanced’ their actual smiles would yield more positive interpersonal outcomes than avatars that conveyed participants’ actual smiles on all three interpersonal measures (i.e., participants’ positive affect, attraction toward their partner, social presence of their partner). We also hypothesized that participants in the ‘enhanced smile’ condition would want to spend more time on the avatar-networking platform (vs. FtF communication) than those in the ‘normal smile’ condition.

Second, we wanted to see if there were benefits of adding realistic facial expressions to an avatar compared to providing an avatar with a narrow range of facial movements (i.e., mouth that only opens and closes). The extant literature suggests that increasing behavioral realism yields a more positive communication outcome [35–37]. For example, von der Pütten and colleagues [37] found that participants felt higher levels of mutual awareness when they spoke with a virtual human that blinked, shifted posture, breathed, and nodded in response to the participant (back-channeling) compared to a virtual human that did not exhibit back-channeling behavior. However, the implications of adding realistic facial expressions to an avatar are yet to be explored. As such, we included an avatar that did not exhibit any facial expressions except for the opening and closing of the mouth (i.e., ‘mouth open-close’ condition) along with avatars that offered either a veridical or enhanced version of the participant’s smile in our study design (i.e., ‘normal smile’ and ‘enhanced smile’ conditions).

In summary, the following hypotheses and research question was proposed:

1. Hypothesis 1a-d: Participants in the enhanced smile condition will feel greater (a) positive affect, (b) attraction toward their partner, (c) social presence of their partner and (d) desire to spend more time communicating via the avatar-networking platform compared to those in the normal smile condition.
2. Research Question1a-d: Does the addition of realistic facial expressions influence participants’ (a) positive affect, (b) attraction toward their partner, (c) social presence, and (d) desire to spend time communicating via the avatar-networking program?

Method

Study Design

The present study adopted a dyadic design with three conditions: Normal Smile, Enhanced Smile, and Mouth Open-Close only. Both participants in the dyad were assigned an avatar with identical smiling conditions (e.g., participants in the enhanced smile condition both had avatars that enhanced their actual smiles), although they were only able to see their partners’ avatars.

In order to determine the extent to which the smile should be enhanced in the Enhanced Smile condition, we ran a pilot study with 23 participants who were not a part of this study. The coefficient values of the smile tracking data were initially multiplied by 1.2 when updating the avatar’s smile in the enhanced smile condition. However, participants provided feedback that the smiling behaviors of the enhanced and normal smile conditions were similar. Thus, we increased the enhancement of the smiling coefficients to 1.5, which did lead to perceptions that the avatars in the enhanced smile condition smiled more than the normal smile condition. More importantly, post-pilot interviews with participants showed that they had not been aware of the smiling manipulation and thought that the conversational flow was natural. Based on these results, we chose to multiply the smiling data by 1.5; manipulation checks that were conducted after the actual study also confirmed the validity of this choice.

In contrast, only the coefficient value that represented the extent to which a mouth was open was used to update the avatar in the mouth open-close condition. To allow for a fair comparison across the three avatars, the avatar in the mouth open-close only condition was designed to exhibit a slight smile. The different avatar representations of these conditions is available in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Avatar representation of ‘normal smile’, ‘enhanced smile’, and ‘mouth open-close only’ conditions.

https://doi.org/10.1371/journal.pone.0161794.g001

Materials

All participants were represented by avatars that matched their (biological) genders. The background of the virtual interaction platform was modeled after the room in which the participants were seated. The features of both the female and male avatars were similar to each other with the exception of the hairstyle (Fig 2). Participants only saw their partners’ avatars, and not their own during the interaction. All female participants were given the same female avatar, and all male participants were assigned the same male avatar.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Neutral state female (left) and male (right) avatars used in study.

https://doi.org/10.1371/journal.pone.0161794.g002

Results

As mentioned earlier, six of the participants were excluded from analysis as they reported that they already knew each other (n = 6). In addition, due to a technological failure, participants in one dyad were assigned to different conditions (n = 2) and were thus excluded from analysis. Finally, the four participants that were thought to have guessed the purpose of the study and one of the participants who later reported not being able to see her partner were excluded from analysis (n = 5), leaving a total of 145 participants (n = 50, 46, and 49 for the normal smile condition, enhanced smile condition, and mouth open-close condition, respectively). Chi-squared tests showed equal distribution of gender (χ² = 3.905, df = 2, p = .14) and ethnicity (χ² = 10.99, df = 10, p = .36) across conditions. Age was also well distributed across the ‘normal smile’ (M = 20.86, SD = 2.17), ‘enhanced smile’ (M = 21.57, SD = 2.08) and ‘mouth open-close only’ (M = 20.35, SD = 3.87) conditions, with no significant differences between conditions (F(2, 142) = 2.26, p = .11). The means and standard deviations for all of the dependent variables by condition can be seen in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Means and standard deviations of dependent variables by condition.

https://doi.org/10.1371/journal.pone.0161794.t001

Data from participant who interact with each other are likely to violate the assumption of independence and is thus inappropriate for ANOVA and regression approaches [47]. Multilevel analysis (also known as mixed-effects model, hierarchical linear model, etc.) “combines the effects of variables at different levels into a single model, while accounting for the interdependence among observations within higher-level units” [48]^p3665. As neglecting intragroup dependence can bias statistical estimates, we used a multilevel model to address this issue (for an example of a study that uses this model, see [49]). We used the ‘nlme’ package in R to conduct a linear mixed-model analysis. We accounted for the random effects that arise from the individual subjects who are nested within dyads and used compound symmetry structure for the within-group correlation structure. The different conditions were dummy-coded and treated as a fixed factor (avatar facial expression) with three levels (i.e., normal smile, enhanced smile, mouth open-close only). Gender was included as a control variable to account for the possible gender differences in smiling behavior and perception [50]. Other studies that study the influence of nonverbal cues have included gender as control variables when they were considered relevant [17], [51]). All analyses below are results from this linear mixed-effects model, with the exception of the impact of avatar condition on the time spent in the avatar-networking program (in this case, we used a multiple regression model that used the dyad as the unit of analysis; this was because partners mutually reached the time decision during their conversation and, as such, there was a single data point per dyad) and the correlational analyses. Results from the linear mixed-effects analyses are reported with an estimate value (b) and significance level (p), which is in line with studies that used multilevel models [52–53].

Correlations Among Dependent Variables

All simple correlations among dependent variables are included in Table 2. Correlation analyses showed that the time spent on the avatar-mediated platform was positively correlated with partner attraction (r = .31, p< .001) and social presence (r = .26, p< .01) and negatively correlated with negative affect (r = -1.78, p< .05). While it is difficult to definitively determine causality from this data alone, these results suggest that participants may have preferred to spend more time on the avatar-mediated platform when they were having a positive experience. Correlation analyses also revealed a positive correlation between PANAS and the results of the LIWC analysis (positive affect: r = .27, p< .001; negative affect: r = .16, p< .05). Furthermore, the percentage of positive words used by participants was significantly correlated with positive perceptions of the communication experience, namely partner attraction (r = .29, p< .001) and social presence (r = .32, p< .001).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Simple correlations among dependent variables.

https://doi.org/10.1371/journal.pone.0161794.t002

Discussion

The present study found that participants who interacted with each other using avatars that enhanced their actual smiles felt more positive affect and a greater sense of being present with their partner (i.e., social presence). Furthermore, when asked to describe their experience using the avatar-networking platform, participants were more likely to use a greater proportion of positive words when they had been assigned to the enhanced smile condition, compared to the other conditions. Another important aspect to note is that the overwhelming majority of the participants were not aware of the smile manipulation.

To the best of our knowledge, this is the first study that examines if altering interactants’ facial expressions via real-time rendering on an avatar can influence one’s emotions, evaluations of their partner, and social presence. While there was a single study that did examine if altering interactants’ facial expressions during a video conference would impact creativity [55], the study’s participants were already acquainted with each other, making it difficult to tease apart the impact of the manipulation. Our study controlled for the familiarity factor by having participants interact with strangers. The results of our study offer strong evidence that enhancing certain facial expressions on avatar representation can influence interactants’ communication experience in virtual environments, at least among zero-acquaintance individuals. These findings support the rich literature on nonverbal cues in that increased smiling led to more positive outcomes [27–28], [56–57]. Our findings have practical implications for designers of virtual environments; introducing subtle changes in facial expressions may lead to a more positive experience between interactants. These findings are consistent with the literature on transformed social interaction [12], which show that virtual environments have the potential to allow people to strategically decouple their actual behavior and performance from their virtual representations to produce hyperpersonal communication outcomes that can be more positive than ‘authentic’ representations.

Contrary to our expectations, enhancing the smiling behavior of avatars did not influence the time participants preferred to spend on the avatar-networking platform over face-to-face communication. Considering the fact that participants evaluated their communication experience more positively for the increased smile condition, this is somewhat surprising. One possible explanation is that participants varied in their motives in selecting a communication modality. For instance, participants’ positive experience with the avatar-networking platform may have led to the desire to continue to communicate with their partner using the virtual platform, but this positive interaction experience could also have led to the eagerness to discover what their partners were like in ‘real life’. In addition, the content of the conversation itself may have also influenced participants’ willingness to switch interaction platforms. For instance, some of the participants may have encouraged their partners to meet in person more strongly than others.

Surprisingly, the normal smile condition did not yield additional benefits over the mouth open-close only condition. While we entertained the possibility that the increased realism of avatars with facial expressions that reflected facial motion tracking data would lead to positive communication outcomes, this was not the case. There are two possible explanations for this finding: first, because the avatar was not very high in terms of photographic realism (see Fig 2), participants may not have needed realistic facial expressions for a better communication experience. Studies suggest that avatars yield the most positive effects when the level of visual realism is congruent with the level of behavioral realism [58]. Second, it is possible that the nonverbal cues provided by the voice and up-and-down mouth movements may have been sufficient for simple interpersonal tasks (e.g., 20 Questions Game) between partners who were not required to see each other again. Previous research that compares audio and video conferencing also shows that adding video does not necessarily improve the communication experience [59].

Limitations & Future Directions

Some limitations of the study need to be addressed. First we only used one type of avatar—one that had moderate levels of visual realism. While we chose this avatar to avoid confounds from ‘the uncanny valley effect,’ [60] wherein people feel negative emotions toward virtual representations that looks near-humanlike, it is possible that the appearance of the avatar will moderate the effects of enhanced smiles. Some studies on the perception of animated characters suggest that people perceived the same facial expressions to be less emotionally intense on more schematic faces (e.g., outline sketch) compared to real human faces [61]. Similarly, people considered artificially enhanced facial expressions to be less odd when the visual representation was a schematic sketch of the human face compared to when it was an actual human face [62]. In our study, the avatars were not very high in photographic realism, as they tended to look closer to a childish cartoon character rather than a ‘real’ human face. This may have added to the pleasantness of the interaction and influenced the potential effect of smiles on participants. Future studies should explore if and, if so, how the avatar’s appearance interacts with enhanced facial expressions.

In addition, our pilot study did not examine the quantitative relationship between the degree of smiling enhancement and perceptions when selecting the enhancement coefficient for the main study. As such, while we were able to select an enhancement coefficient that increased perceptions of smiling and still allowed the facial movements to appear natural (i.e., not manipulated), we cannot rule out the possibility that selecting another smiling coefficient through more quantitative testing may have yielded stronger effects. This is beyond the scope of the present study, but future studies should elucidate the relationship between the extent of smiling enhancement and perceptions.

Second, we focused specifically on zero-acquaintance dyads. While we did this to focus specifically on the effects of enhancing the avatar’s smile, it is possible that transforming nonverbal behaviors may have different implications depending on the relational dynamics between communicators. While they did not study this research question, [63] found that people smiled and laughed more and rated an audiotaped stand-up comedy routine more positively when they listened to canned laughter that they thought came from members of their in-group rather than their out-group. Studies have also shown that perceived group membership of the source influences people’s mimicry of nonverbal cues and empathy [64–65]. Future research could thus explore the boundary effects of enhancing smiles in virtual reality depending on the characteristics of the interactants’ relationship.

In addition to partners’ relationships with each other, the context of the conversation may also influence the effect of smiles. In the present study, participants were asked to play a 20 questions game and perform ‘Getting to Know you’ task in order to facilitate communication flow and allow for some control over the communication content. However, the relaxed and positive nature of a game may have yielded more smiles or fostered higher levels of attraction than a more serious task may have. Future studies should consider how the communication context can limit or enhance the influence of an avatar’s facial expressions.

In addition, our study was conducted in a social context, wherein participants were generally friendly towards one another. However, the increased smiles may be interpreted differently if the context was more hostile or competitive. In this case, enhanced smiles on an avatar may be perceived to be inconsistent with the interaction partner’s other behaviors (e.g., verbal), leading to poor communication outcomes. Studies have shown that people respond negatively to nonverbal behavior that is incongruent with verbal behavior for both real people and digitized agents, lending some support to this conjecture [66–67]. As such, more research is needed to determine when augmenting certain facial expressions yield positive outcomes, and when this is not the case.

It is also worth noting that the concept of social presence has often been linked to feelings of empathy. For instance, [68] proposed that empathy is a factor worthy of consideration in the psychological involvement dimension of social presence. The perception of being in the same space as another person has the potential to allow us to feel what that individual feels when something happens to her or him [69]. Virtual reality has the potential to allow people to step in the shoes of those whom they have not met before, or experience the lives of members of marginalized social groups. Future research can explore the links between avatar facial expressions, social presence and empathy, and examine how various aspects of virtual reality can contribute to a greater level of empathy among individuals.

Finally, our study participants were university students, and are thus more likely to have had experience interacting within virtual environments. However, as some studies suggest that age is associated with the level of social presence people feel in a virtual environment [70], it may be necessary to consider how different age groups as well as individuals with less experience with virtual environments respond to the different avatars before we are able to generalize our findings.

Conclusion

The current results offer cogent evidence that enhancing the smile on one’s avatar can lead to more positive outcomes compared to when the smile is accurately mapped on the avatar. This is the first study to explore the outcomes of altering participants’ actual facial expressions in real-time on a computer-generated avatar using both self-report and behavioral measures. While the facial expression manipulation did not influence the time participants chose to spend on the avatar-networking platform or their smiling intensity, both of these measures were correlated with participants’ perceptions of their partners and social presence. This suggests that behavioral measures in virtual environments can be used as an unobtrusive measure that can predict people’s attitudes.

Supporting Information

Acknowledgments

The authors would like to thank Dana Murphy for her contribution to this work in terms of programming assistance. In addition, we would like to thank Jack Klein, Kristina Bassi, Maggie Murphy, Max Wolff, Michael Blaney, Roxanna Reaves, Sophia Pink, and Tara McCullough for their assistance in running study participants. Academic licenses to the software used in the study were donated by Faceshift. This project was funded in part from an academic research gift from Google.