Learning motion primitives and annotative texts from crowd-sourcing

The full-body motions of humans were measured by optical motion capture or wearable motion sensors. Position data at each point on the body were converted into motions of a computer-generated model character using inverse kinematic calculations. Videos of these motions were made viewable on the Internet. Figure 1 shows examples of frames from the videos.

The task of manually assigning descriptive annotation to each motion video was carried out via crowdsourcing. In the annotation task, a video, a playback time, and a word representing the subject are presented. The user inputs descriptive text in English corresponding to the motion initiated by the given subject at the specified time. Using this task, a training dataset of motions and corresponding descriptive texts can thus be collected. In this study, the annotation task was openly available from our research laboratory’s website as shown by Figure 2. The students and researchers from my department are allowed to annotate the motions such that appropriately assigned descriptive texts can be collected efficiently.

The task described above provides descriptive sentences and their corresponding times. This task does not provide a start point and an end point of a motion segment to which the descriptive sentence is assigned. I manually detected the start point and end point for each motion segment after the annotation task, and consequently obtained datasets of the motion segments and their descriptive sentences.

A human full-body motion is represented by a sequence of angles of all the joints. Each sequence is encoded into an HMM λ. An HMM is a statistical model used to classify input data into an appropriate category. An HMM is defined by a compact notation λ={Q,A,B,Π}, where Q={q ₁,q ₂,⋯,q _n } is the set of nodes, A={a _ij } is the matrix whose entries a _ij are the probability of transitioning from the ith node to the jth node, B is the set of output probability density functions at the nodes, and Π={π ₁,π ₂,⋯,π _n } is the set of initial node distribution. In this study, the parameters of the HMM are optimized by Baum–Welch algorithm using its corresponding sequence of the joint angles. Baum–Welch algorithm is one of the expectation maximization (EM) algorithms [16]. The motion can be classified into its relevant HMM that is the most likely to generate this motion. The motion is expressed by the discrete form of the index of the HMM, and the HMM is hereinafter referred to as a “motion symbol”.

In the annotation task, a descriptive annotation is assigned to each motion symbol. Consequently, a training dataset of motion symbols and descriptive texts is collected. More specifically, each training data is a pair of motion symbol λ _k and a descriptive sentence ω _k , where the descriptive sentence is expressed by a sequence of l _k words, \(\boldmath {\omega }_{k} = \left \{ {\omega ^{k}_{1}}, {\omega ^{k}_{2}}, \cdots, \omega ^{k}_{l_{k}} \right \}\). This paper proposes a statistical model that converts the motion symbol to descriptive sentences as shown by Figure 3 [12]. This conversion results in understanding human full-body motion in the forms of sentences. The statistical model consists of two modules. One module learns the probabilistic relations between a motion symbol λ and a word ω. This module is hereinafter referred to as “motion language module”. The other module learns the probabilistic relations of transition of two words in a sentence. This module is referred to as “natural language module”.

Figure 4 shows an overview of the motion language module that consists of three layers. The top layer includes motion symbols, the middle layer includes latent states, and the bottom layer includes words. A motion symbol generates a latent state, and a latent state generates a word. Association between the motion symbols and the words are represented by a generative model. Probabilistic relation between the motion symbol and word is represented using the probability P(s|λ) that the motion symbol λ generates the latent state s, and the probability P(ω|s) that the latent state s generates the word ω. These probabilities are optimized such that the total probability that motion symbols generate the words in the descriptive sentences in the training dataset is maximized. The logarithm of the total probability is written as

where θ is a set of the probabilities P(s|λ) and P(ω|s). I assume that a word is independent of each other, and is dependent on only the motion symbol in the motion language module. Equation (2) can be subsequently rewritten as Equation (3). The dependence relationship between two words is learned by a natural language module. The The optimal θ is derived by the iterative computation. Let θ ^[t] be the set θ derived at t-th iteration. The probabilities P(ω,s|λ), P(s|λ), and P(ω|s) derived at t-th iteration are rewritten as P(ω,s|λ,θ ^[t]),P(s|λ,θ ^[t]), and P(ω|s,θ ^[t]) respectively. Equation (4) at t-th iteration is rewritten as

where E _P [R] denotes the expected value of R given the distribution P. According to Jensen’s inequality, Equation (7) satisfies the following relation.

Using Equation (3) and Equation (8), the following equations can be derived.

Equation (11) represents the Kullback Leibler information that measures the dissimilarity between the distributions \(P(s | \lambda _{k}, {\omega ^{k}_{i}}) \) and \( P(s | \lambda _{k}, {\omega ^{k}_{i}}, \theta ^{[t]})\). The Kullback Leibler information becomes zero only when these two distributions are exactly same, and takes a positive value otherwise. The difference between Φ(θ ^[t+1]) and Φ(θ ^[t]) is subsequently written as follows:

The distribution \( P(s | \lambda _{k}, {\omega ^{k}_{i}})\) is assumed to be estimated as \(P(s | \lambda _{k}, {\omega ^{k}_{i}}, \theta ^{[t]})\) based on the motion language model derived at t-th iteration, and the third and fourth terms in Equation (12) take a positive value and zero respectively. Hence, I only have to search for θ ^[t+1] such that the first term in Equation (12) becomes greater than the second term because the incremental update of θ ^[t+1] increases the total probability Φ of the training data. More specifically, the first term only has to be maximized by θ ^[t+1]. Using the probabilities P(s|λ,θ ^[t+1]) and P(ω|s,θ ^[t+1]), This maximization can be reduced as follows

where the terms independent of θ ^[t+1] are eliminated. The probabilities P(s|λ,θ ^[t+1]) and P(ω|s,θ ^[t+1]) are constrained as follows:

By applying the method of Lagrange multiplier to Equation (13), the probabilities P(s|λ,θ ^[t+1]) and P(ω|s,θ ^[t+1]) at t+1-th iteration can be analytically derived.

where n _k,i is the number that the word ω _i appears in the sentence ω _k assigned to the motion symbol λ _k . Note that ω _i denotes the i-th word in the set of words, and \({\omega ^{k}_{i}}\) denotes the word at the i-th position in the sentence assigned to the k-th motion symbol. The processes described above are iterated, and consequently the optimal probabilities P(s|λ) and P(ω|s) can be derived.

Figure 5 shows an overview of the natural language module. This module extracts the probability π(ω) of starting at the word ω and the probability P(ω _j |ω _i ) of transitioning from the word ω _i to the word ω _j using a training dataset of sentences assigned to the motion symbols. The probabilities π(ω _i ) and P(ω _j |ω _i ) are optimized such that the probability that the natural language module generates the training sentences. The logarithm of this probability is expressed by

where 𝜗 is a set of probabilities π(ω) and P(ω _j |ω _i ). The optimal 𝜗 can be analytically derived as follows.

where c(ω) is the frequency of the sentence starting at the word ω, and c(ω _i ,ω _j ) is the frequency of transitions from the word ω _i to the word ω _j .

The conversion from the motion symbol \(\lambda _{\mathcal {R}}\) to its descriptive sentences \(\boldmath {\omega }_{\mathcal {R}}\) can be treated as the problem of searching for the sentences that are most likely to be generated by the motion symbols. This problem is expressed as follows:

where \(P(\hat {\omega }_{1}, \hat {\omega }_{2}, \cdots, \hat {\omega }_{l} | \lambda _{\mathcal {R}})\) is the probability that the motion language module generates a set of words \(\left \{ \hat {\omega }_{1}, \hat {\omega }_{2}, \cdots, \hat {\omega }_{l} \right \}\) from the motion symbol \(\lambda _{\mathcal {R}}\), and \(P(\hat {\boldmath {\omega }} | \hat {\omega }_{1}, \hat {\omega }_{2}, \cdots, \hat {\omega }_{l})\) is the probability that the natural language module arranges the set of words \(\left \{ \hat {\omega }_{1}, \hat {\omega }_{2}, \cdots, \hat {\omega }_{l} \right \}\) into the sentence \(\hat {\boldmath {\omega }}\). Therefore, these two probabilities can be written using the probabilities defining the motion language module and the natural language module.

where \( P(\hat {\omega _{i}}| \lambda _{\mathcal {R}})\) can be calculated as \(\sum _{j} P(\hat {\omega _{i}}|s_{j})P(s_{j}| \lambda _{\mathcal {R}})\). Substituting Equation (24) and Equation (25) into Equation (23) and taking the logarithm of it, Equation (23) can be reduced to the following equation.

Equation (26) can be efficiently solved using Dijkstra’s algorithm.

An experiment on the conversion from the full-body motions of human to the descriptive sentences was conducted by using our proposed statistical framework. The full-body motions were measured using an inertial motion capture system where 17 IMU sensors were attached to a human performer. This measurement was conducted with the approval of the ethical committee of the University of Tokyo. Positions of 34 selected bodied part in the human full-body in the trunk coordinate system were derived via kinematic computation using a human figure model with 34 degrees of freedom. Each measured motion segment is encoded into an HMM. The HMM consists of 30 nodes, each of which has one Gaussian distribution, and the type of node connection is left-to-right. A descriptive sentence is manually assigned to each HMM via crowdsourcing. In this study the full-body motions of one performer were measured during working at the office or giving a lecture, and 621 motion symbols, each of which a sentence is assigned to by five users, were subsequently collected. The number of different words used in the descriptive sentences was 419. Table 1 shows sample parts of the training dataset of motions and their descriptive sentences.

After learning the motion language module and the natural language module using the training dataset as shown by Table 1, the proposed framework was tested on 100 different full-body motions of human. Each motion is converted to five descriptive sentences that are most likely to be generated by both the motion language module and the natural language module. Figure 6 shows the experimental result of conversion from a full-body motion to sentences, where a sentence containing less than three words is removed as a candidate sentence. A motion “sitting” is converted into sentences “a person sits”, “a person sits down”, “a person sits back” and “he sits down”. A motion “drinking” is converted into sentences “a person is drinking” and “he is drinking”. These sentences were confirmed to correctly represent the full-body motions. A motion “sitting with legs crossed” is correctly converted into sentences “he is sitting”, but it is wrongly converted into another sentence “he is sitting with his legs”. Additionally, it is correctly converted into a long sentence “he is sitting with his legs crossed”, that is ranked lower than the wrong sentence “he is sitting with his legs”. A motion “writing on a blackboard” is converted into a correct sentence “he is writing”, and wrong sentence “he is writing on” and “he is writing a blackboard” that are close to the correct sentence “he is writing on a blackboard”. The several wrong sentences are terminated at the inappropriate words, and longer sentences are unlikely to be generated. The natural language model needs to be extend to word trigrams such that it represents the relations among words that are distant from each other in the sentences, and the conversion from the motion to the sentence, expressed by Equation (26), should be modified to take into account the length of sentences.

I also quantitatively evaluate the conversion from the motions to sentences. Five users assigned a descriptive sentence to each test motion. The performer and users in this test phase are same as those in the learning phase. Each motion that was converted to several candidate sentences, one of which is exactly same as the sentence assigned to this test motion was counted as the correct. The accuracy of the conversion can be computed as a ratio of the correct motions to the test motions. The number of the candidate sentences was varied. In the case that the number of the candidate sentences was set to 1, the accuracy of the conversion was 0.34. The number of the candidate sentences was set to 2, the accuracy of the conversion reached 0.59. Three, four, and five candidate sentences result in the accuracies of 0.64, 0.68 and 0.71 respectively.