RMoCap: an R language package for processing and kinematic analyzing motion capture data

Kinematics is the branch of mechanics that deals with the motion of the bodies and system without considering the force [28]. Forward kinematics is specified by the joints parameters and kinematic equations that are used to compute the position of the end effector from specified value for each joint parameter (in this paper, we do not take into account an inverse kinematic problem which is not a case in MoCap technology). Let us define the Direct Kinematic (DK) model/description as motion notation in which body joints positions are directly described by 3D coordinates of each body joint (no further calculation is required to determine they spatial positions). This model does not take into account rotation, so it treats each body joint as a material point in space rather than a rigid body. In Hierarchical Kinematic (HK) model/description, we define the motion as a set of 3D rotations of body joints that are organized in hierarchy (so called kinematic chain)—a tree structure that has a root joint. Distances (offsets) between body joint and its parent are defined relatively to the parent joint. Also rotations are defined relatively to the parent joint. The root joint, that does not have the parent joint, beside rotation also contains information about translation of the whole body. An example visualization of joints organized in hierarchy can be seen in Fig. 1. In that figure, a root joint is a Hips joint and it has three children: SpineLow, RightThigh and LeftThigh; RightThigh has a child RightLeg, etc. To get spatial coordinates of body joints from HK model, we have to recalculate it to DK. DK model is often more useful for kinematic data analysis. However, for example, computer graphic often utilizes HK, because it has the information about joints orientation (rotation). Cheap MoCap hardware like Kinect generates data in DK model. To get joints orientation, we need to recalculate DK to HK model. To do so, we need to have a prior definition of joints hierarchy that is compatible with DK model (that has same number of joints).

Nearly, each motion capture hardware manufacturer supplies users with a basic graphical interface and/or application programming interface that enables conversion of a raw data into popular data formats that can be processed by third-party software. Also, there are often dedicated packages for computing basic kinematic parameters like time of motion, trajectory length, linear velocity, acceleration, angles between body joints, etc. Those parameters are often used by scientists to generate tabularized values that later become reference values for various motion activities [16, 17, 33]. From the statistical and computer science perspective, those are very basic operations. Those statistic can be generated by several common programming language instructions (for example summary command in R language). Due to this, our package does not implement operations that are available as generic R language functions and we will not discuss those approaches later on.

Among the most interesting and fruitful methods of motion analysis are motion capture averaging methods and motion path matching approaches. The motion averaging is a process in which as an input data we take several recordings of a person who performs same action. After applying averaging, we generate a single recording that should maximize value of similarity measure between averaged recording and each input recording (so motion averaging is an optimization procedure). The aim of motion averaging is to remove small, random noises that might be present in MoCap recording. Motion averaging does not remove systematic errors that might visualize some common motion inaccuracies. Depending of application of MoCap analysis, an inaccuracy might be an effect of some disabilities caused by illness (medical and rehabilitation application) or wrong performance of technique (sport application). Our package implements method presented in paper [12] that enables averaging 3D MoCap data using dynamic time warping barycenter averaging (DBA) [24]. It operates in quaternion space using Markley averaging approach instead of barycenter [19]. Package RMoCap is the first open-source release of this method.

The motion path matching approaches analyze trajectory of certain body joints and align one MoCap data to another to find some important differences between them. There are many papers that describe application of that type of analysis in 1D and 2D [7, 21, 22, 29‐31]. In our package we implement 3D trajectory comparison method based on Dynamic Time Warping (DTW) approach that is able to find and visualize the highest local differences in kinematic chain.

The IMU-based (Inertial Measurement Unit) MoCap hardware, that typically contains accelerometers, gyroscopes and magnetometers, is capable to measure the acceleration of the objects that are attached to costume. By numerical integration, we can estimate the distance that those sensors moved starting from the known, initial position. Due to limited precision of IMU (both static and dynamic), some unavoidable drifts from the ”real” body position are continuously introduced. However, the most important problem with IMU-based MoCap is the fact, that those three earlier mentioned measuring sensors (accelerometer, gyroscope and magnetometer) are not enough to measure the body displacement is 3D space relatively to the ground (we will call it later a body displacement). The only reliable displacement returned by those sensors is relative motion to root joint of HK model. To measure body displacement, some additional sensors have to be introduced, for example, vision-based (cameras) or pressure sensors located on feet. Vision-based system is not reliable as long as it is not a professional MoCap with at least the same price as IMU system itself. Pressure sensors operate under some heuristic that often introduces errors in estimation body displacement vector. Knowing that the RMoCap package introduces function that is a heuristic that can estimate the body displacement of MoCap data. The proposed heuristic is a novel extension of the method proposed in [10]. The extension in RMoCap package makes method capable to process data that do not contain information about acceleration.

In this subsection, we will only discuss package for MoCap data analysis. There are dozens of open packages for converting 2D image data into 3D MoCap (for example [5]) and for MoCap-based pattern recognition [9]; however, they are designed to solve different problems than our algorithms. Also kinetic (not kinematic) analysis performed, for example, by MotionLab is out of our scope [25].

Our package is capable of freely converting not only HK into DK model (there are many examples of that in number of programming languages) but also DK into HK which is a more complex optimization task. For our best knowledge, this is a first open-source publicly available package that implements this type of conversion.

R language has very little packages directly dedicated for MoCap data analysis. As far as we know, there is only mocap [27] package that enables loading ASF/AMC files and converting them into a list data structure. The visualization functions of that package seem to be based on old version of third-party libraries and are not operational any more. The second package we found, mocapGrip [14] encapsulates a python motion capture project dedicated for annotations analysis, which is utilized mostly in linguistic and psychology. Both of those packages do not contain functions that can be helpful in motion processing and kinematic analysis. DBA algorithm is implemented in R language package [26] and DTW in package [8]; however, both of those implementations do not allow to use quaternion data. Also, DTW implementation is not flexible enough to access all algorithm parameters that we need. Due to this, RMoCap has its own implementations of both algorithms, independent from both packages.

Among important open-source packages created in other than R programming languages is Mokka,¹ which is a motion capture kinematic and kinetic analyzer with graphical interface that implements basic plots, media integration and data importing functions. It utilizes functions from an open source framework Biomechanical ToolKit (BTK) [2]. BTK supports C++, Matlab and Python programming languages. OpenSim is a software to create and analyze dynamic simulations of movement [4]. An open source toolbox MoBILAB [23] can be used for analysis and visualization of mobile brain/body imaging data. HuMAns toolbox [32] includes a biomechanical model of a complete human body and proposes a set of versatile tools for modeling, capture, analysis and simulation of human and humanoid motion (it is an open-source software, distributed under the GPL License). MoCap Toolbox [3] is a set of functions written in Matlab for analyzing and visualizing motion capture data. It covers basic visualization and analysis approaches, such as general data handling, creating stick-figure images and animations, kinematic analysis (mean and standard deviation of velocity and acceleration), and performing Principal Component Analysis (PCA) on movement data.

Although all mentioned software contain useful methods, they do not cover new functionalities that are included in RMoCap that we initially presented in Sect. 1.1. All of them will be discussed in detail in the rest of this paper.

All algorithms that are implemented in RMoCap package are described using standard mathematical notation or, if this notation might be too clumsy, with code in R language or pseudocode similar to R language. In case of mathematical notation, matrices are named with capital letter, numbers are in lowercase, angles are lowercase Greek letters and vectors are lowercase letters with arrow. In R/R pseudocode, all fields of complex data types (i.e., named columns of lists, matrices, data frames) are accessed with $ operator. Vectors are defined with parentheses, while accessing object with certain index in data frame, list or matrix require square brackets. Dot symbol ”.” might be a part of the name of the variable. Hash symbol ”#” begins a single-line commentary. We use notation ”/* */” for commentaries in pseudocode.

RMoCap package contains several MoCap recordings that can be used to test algorithms we have implemented. The MoCap data were recorded using Shadow 2.0 wireless motion capture system, which is high-end professional IMU-based solution. We use this hardware for our everyday work. More about recording process can be learned from [12]. Motion recordings are example karate techniques performed by a world and national class professional (medalists) karate athletes. There are both short recordings of single karate kicks and also longer karate kata motion sequences. Of course, our algorithms can read and operate on other recordings. There are two basic file formats RMoCap uses: Biovision Hierarchy file format (BVH), that is a very popular format to hold HK model [20] and also comma-separated vector (CSV) file format. Requirements for CSV are proper naming of columns:In case of R language, CSV files are often stored in memory in data.frame structure. If data.frame columns have same number of joints definition in DK model as HK in BVH file, data can be easily converted from DK into HK (see example in Sect. 3). Most of the current MoCap systems and motion repositories store data in at least one of those formats. It requires basic knowledge of R to prepare a data.frame imported from CSV to work with RMoCap.

The R package RMoCap [11] is distributed under the GPL-3 license. It depends on the R packages smoother v. 1.1 [13], rgl v. 0.99.16 [1], RSpincalc v. 1.0.2 [6], subplex v. 1.5.4 [15], signal v. 0.7.6 [18], compiler v. 3.5.0.

The R package RMoCap is hosted by GitHuba² and its dependencies are available at https://​CRAN.​R-project.​org/​ and can be installed as follows:

The method of finding n-dimensional rotation matrix \(R_{n}\) that rotates vector \(\overrightarrow{x}\) to \(\overrightarrow{y}\) is not a basic but already known algebraic procedure. Let us assume that \(\overrightarrow{x}\) and \(\overrightarrow{y}\) are linearly independent. If this condition is false, the rotation is represented by identity matrix.

At first we will generate 2D rotation matrix \(R_{2}\) such that: \(x\cdot R_{2}\) is linearly dependent to y.where \(\alpha\) is an angle on the plane between \(\overrightarrow{x}\) to \(\overrightarrow{y}\), \(\cos (\alpha )= \frac{\overrightarrow{x}\circ \overrightarrow{y}}{\left| \overrightarrow{x} \right| \cdot \left| \overrightarrow{y} \right| }\), \(\sin (\alpha )= \sqrt{1-\cos ^{2}(\alpha )}\) and \(\circ\) is a dot product.

In the next step, we will generate orthonormal pairs of vectors \(\overrightarrow{a}\), \(\overrightarrow{b}\) from \(\overrightarrow{x}\), \(\overrightarrow{y}\) using Gram–Schmidt process.

We need to normalize vector \(\overrightarrow{x}\):and then by projection of vector \(\overrightarrow{y}\) onto the line spanned by \(\overrightarrow{a}\)The projection matrix on subspace designated by unit vectors \(\overrightarrow{a}\), \(\overrightarrow{b}\) is:The projection matrix onto complemented subspace n-2 subspace is:where \(I_{n}\) is n – dimensional identity matrix.

Because the rotation happens in subspace designated by vectors \(\overrightarrow{a}\), \(\overrightarrow{y}\) we can get n-dimensional substitute of \(R_{2}\) by changing the base of \(R_{2}\) to be n-dimensional and by completing it with \(P'\):Knowing how to calculate \(R_{n}\), we can go to the first step of DK to HK mapping that is determination of rotation of root joint.

Among all joints in HK model, the root joint is a special case because of two factors:Let HK be a MoCap data structure in hierarchical model, while DK a MoCap data structure in direct kinematic model. Calculation of translation data is straightforward (see below pseudocode):

Finding rotation of the root joint is, however, more complex. At first we need to find any two direct descendants of the root. If there are more than two, the rest is irrelevant for this algorithm.

We choose one of these descendants (Child1) and calculate rotation matrix that rotates vector designated by offset of HK root joint onto vector designated as difference between Child1 and Root joints coordinates in DK model.

Where vector.to.unit is a function that normalizes the vector and rotation.matrix is a function that implements an algorithm from Sect. 3.1.

Having rotation matrix Rx2y, we can now calculate quaternion after rotation by which we will rotate one vector onto another. In R, we can do it with function DCM2EA from package RSpincalc. Pseudocode 2 finds matrix that maps vector defined by an offset of root joint onto vector designated by position of child joint. The rotation matrix Rx2y as it was said in Sect. 3.1 operates on the 2D plane and there is no guarantee that the initial solution we found also maps all other direct descendants of the root joint. That is because the solution we initially obtained might be rotated around \(\overrightarrow{map}\). To find the appropriate 3D rotation that also maps other children of root we have to find additional rotation that applied after the first one correctly aligns both child-parent vectors. To do this, we can apply the following algorithm:

Optimization from Pseudocode 3 can be done with Simplex method that is implemented for example in function subplex from package subplex. Functions EV2Q that calculates quaternion from axis angle is from package RSpincalc. The variable HK in last line of above pseudocode contains correctly mapped root joint.

After processing the root joint, we iterate through all its children. Those children become parents of the following algorithm. All joints that have parent joint (joints with children and end sites) are processed in the same way. Because we want to find rotation angles relatively to the parent joint, we need to know the transformation matrix Trans of the parent joint. Trans is calculated in the same way as it is in mapping procedure from HK to DK model (see Sect. 2). See Pseudocode 4 for more details.

As the result in variable Rx2y.n, we have the rotation matrix of analyzed joint. After processing a parent joint (Parent.n), we find all children of this joint. Those joints are processed with Pseudocode 4. The program ends when there are no more joints in HK model.

The following R code generates HK from DK and creates plot presented in Fig. 3. As can be seen Euler angles may contains nonlinearities (near 76 and 100 sample) that are present due to periodic domain of trigonometric functions; however, the effector trajectory remains smooth.

The more advanced examples are presented in Supplements—R replication code.

This approach has three main assumptions:At first we calculate magnitudes of acceleration of right \(M_\mathrm{{R}}\) and left foot \(M_\mathrm{{L}}\) and we smooth them with Gaussian kernel. \(M_\mathrm{{R}}\) and \(M_\mathrm{{L}}\) are vectors that are obtained as a result of smoothing right foot \(\overrightarrow{RightFoot.a}\) and left foot \(\overrightarrow{LeftFoot.a}\) acceleration signals, respectively.where \(\overrightarrow{RightFoot.a}\) is an acceleration signal (a vector of 3D acceleration vectors), \(G_{\sigma }\) is a Gaussian kernel and \(\bigotimes\) is convolution operator.

In Fig. 4, we present acceleration magnitude plot obtained for MoCap data from Heian Shodan motion sample from RMoCap package. It can be clearly observed which foot is one with higher magnitude of acceleration (this foot moves) moves and which one with lower magnitude of acceleration remains on the ground.

In the next step for each sample i of \(M_\mathrm{{R}}\) and \(M_\mathrm{{R}}\), we check if \(M_\mathrm{{R}}[i]> M_\mathrm{{L}}[i] \wedge M_\mathrm{{R}}[i+1] > M_\mathrm{{L}}[i+1]\). If it is true, there is a motion of right foot in this sample and left foot remains on the ground. We perform displacement correction of all \(\overrightarrow{ALL.Dxyz}\) displacement vectors (of all body joints vectors) in our dataset:where \(i:samples.count(\overrightarrow{LeftFoot.Dxyz})\) is a range of samples to be corrected starting from \(i\)th sample and ending on the last index of sample from the MoCap dataset. \(\overrightarrow{LeftFoot.Dxyz}[i]\) is an \(i\)th sample vector of left foot displacement.

The second condition we take into account is \(M_\mathrm{{R}}[i]< M_\mathrm{{L}}[i] \wedge M_\mathrm{{R}}[i+1] < M_\mathrm{{L}}[i+1]\). If it is true, there is a motion of left foot in this sample and right foot remains on the ground. We perform displacement correction of all \(\overrightarrow{ALL.Dxyz}\) displacement vectors (of all body joints vectors) in our dataset:In the next step, we need to correct the vertical (Y) coordinate of tracked person so that foot that remains on the ground would be on the same ground level GroundPosition.Dy, which might not be a true condition in original dataset (see Fig. 5a). Let us assume, that initial position of feet is correct (both feet are on the ground level). For each motion sample i of \(M_\mathrm{{R}}\) and \(M_\mathrm{{R}}\) we check if \(M_\mathrm{{R}}[i] \geqslant M_\mathrm{{L}}[i]\). If it is true, there is a motion of right foot in this sample and left foot remains still on the ground. We perform displacement correction of all ALL.Dy displacement vector coordinates (of all body joints vectors) in our dataset:If \(M_\mathrm{{R}}[i] < M_\mathrm{{L}}[i]\) the processing is analogical to above.

The overall results for large motion sample are visualized in Fig. 6. Red silhouettes are original while green are corrected by our approach. An example R code that utilizes calculate.kinematic and plots results analogical to Figs. 4 and 5 is presented below.

In this approach of motion direction correction, we indicate which body joint should be immobile (to have constant position) during the whole motion. We might want to do it in two main scenarios: we do not have acceleration data or/and we want to analyze motion relatively to some body joint. After indicating the constant joint we apply substantially the same approach as in (15); however, instead of \(\overrightarrow{LeftFoot.Dxyz}\), we use displacement vector of the chosen joint. The application of this method will be discussed in section about Motion analysis in RMoCap package.