Facial expressions and speech acts: experimental evidences on the role of the upper face as an illocutionary force indicating device in language comprehension

See the pioneering work by Kendon (1995) on the role of gestures as illocutionary markers in southern Italian conversation. On gestures as illocutionary markers see also Kendon (2004).

Much work has been done also in the field of robotics and of human/robot interaction; see, for instance Granström, House & Beskow (2002).

On the idea that humans express and recognize emotion on the basis of multiple context-sensitive modalities where speech and facial expressions constitute two relevant clues that often interact in this process see also Emerich et al. (2009).

Sadock (1974) has provided a general scheme aimed at explaining the structure of illocutionary acts. According to Sadock, human language is characterized by three crucial functions: representing the world, altering society, expressing emotions. On the basis of this assumption, he argues that identifiable in every language are three distinctive sentence types that are associated with these constitutive functions: the declarative type, the interrogative type, and the imperative type. Speech acts can therefore be classified into three main categories: assertions, questions, and requests.

To be noted is that in its spoken form, English can sometimes be as flexible as Italian in the interpretation of the modality of an utterance; e.g., "Marco’s studying!" vs. "Marco’s studying?".

That in Italian the three types of speech act can be uttered by the very same phonological sequence is also due to a phonological ambiguity (homophony) of the verb studia, which means both a third person indicative (Marco studies) and a second person imperative (Marco, study!).

The fact that Italian interrogative and imperative clauses can exhibit a linear word order that is prima facie identical to that of declarative clauses is a consequence of three well-known syntactic properties of Italian: (1) overt movement of the inflected lexical verb to C(P), (2) subject pro-drop, (3) a rich clausal “Left Periphery” (Rizzi, 1997). Since (1) V rises to a position in the CP domain in overt syntax and (2) the subject position (Spec, TP) can be occupied by a silent pro, as soon as (3) an optional DP is merged higher up in a topic position, the linear order will be “(DP) – V – pro,” with pronunciation of only the DP and the V at spell-out.

It might be argued that a way to develop an expectation about the compatibility of different AUs with different illocutionary forces would be to look at the emotions that the seminal work of Ekman & Friesen (1978), is typically associated with the AUs and those that are considered compatible with the different illocutionary forces under examination. The reason why it is hard to elaborate such an expectation, however, is that the fact that a certain illocutionary force entails a particular emotion (e.g., an order might express anger) does not directly imply that an AU conveying the same emotion can be considered compatible with this illocutionary force. As shown by Shah et al. (2013), in fact, emotions recognition that jointly uses facial expressions and speech represents a controversial issue: if, on the one hand, talking provides clues about the emotional state of the speaker, on the other, it distorts her expression of emotion on the face. Shah et al.’s experimental results show in particular that emotion prediction in the presence of speech and facial AUs is less accurate when the person is talking than it is in a silent condition. In other words, the emotion conveyed by the joint production of a certain illocutionary force with a particular action unit is expected to be the result of an interaction between the two modalities instead of a mere sum of the two components.

According to Searle (1975), they possess different preparatory conditions.

Heritage (2012), however, has argued that the questioner is not on a par with, but on a lower (epistemic) level than, the question addressee.

We are aware that, although the content conveyed by three different illocutionary forces was realized with the very same linguistic form (e.g., “Marco studia”), there might have been an effect of the semantic content of the different target sentences in the production of the speech act. A sentence conveying a more impositive content like, for instance “You are on punishment,” although its Italian translation can convey assertions, questions, or orders with the same linguistic form (i.e., Ita. Tr. “Sei in castigo”) is usually expected to be interpreted as an order (even if clearly other possible interpretations are compatible). However, we decided to avoid controlling the variable of the content of the sentences used in our experiment for two reasons: (1) in a first exploratory experimental study we preferred to leave the item as ecological as possible; (2) we intentionally admitted a potential interference of the semantic content in order to verify whether there was an effect of the illocutionary force type on the AU production even in cases where the propositional content expressed by the sentence prompted the association of a certain illocutionary force. However, this effect was controlled by including the content of the sentence as a random effect in statistical models.

During the development of the experimental material, the FACS coder who helped the actors in production of the pictures representing the different AUs noted a substantial difficulty in the production of AU1, which seems to be an AU type requiring a specific training in order to be produced voluntarily.

AU2 was produced unilaterally for the very same reason.

As filler illocutionary types we distributed two tokens across the items for each one of the following types of illocutionary forces: Advice, Vow, Bet, Felicitation, Wish, Recommend, Agree, Exhortation, Promise, Announcement, Compliment, Concession, Supplication, Plea, Appreciate, Deduction.

To be noted is that in spoken language many aspects like the tone of voice and prosody contribute to the understanding of the illocutionary force of a speech act. In our research, we decided to use linguistic stimuli in a written form in order to investigate whether specific facial expressions are associated with specific illocutionary forces independently of prosody or other verbal or non-verbal IFIDs. Further research should be conducted in order to determine whether different prosodic realizations associated with a certain facial expression may affect the interpretation of the illocutionary force of an utterance. We thank an anonymous reviewer for this suggestion. It should also be noted that our study concerns the relation between facial expressions and illocutionary forces and not between facial expressions and sentence modality. In other words, we deal with the interpretation of “utterances” not of “sentences.” For this reason, although the items were presented in a written form, the various scenarios were presented as fictional ordinary verbal interactions, and participants were required to evaluate the different sentences as if they were uttered in a real verbal interaction.

This is an interpretation of the results about AU4 + 7 that requires further investigation. Research on the relation between emotions and illocutionary forces is in fact still on the way to be conducted. Moreover, there is disagreement about what kinds of emotions are conveyed by different AUs.

According to Brown and Levinson (1987), a face-threatening act is an act that threatens either the speaker’s face or the hearer’s and they may regard either the positive or the negative face.