Automatic analysis of dialect/language sets

This section considers traditional MFCCs as a means to represent spectral based differences between dialects. Since we pursue a statistical approach, extracted features are modeled using GMMs. The proposed method of assessing proximity is based on comparing the log likelihood score statistics. An earlier version of this method was presented by Mehrabani and Hansen (2008). First, MFCC features are extracted from the available audio data. Each dialect is modeled using traditional GMM training. In this study, 64-mixture GMMs and 12-dimensional MFCCs (excluding the 0th cepstrum coefficient) were used. The same number of mixtures was applied to model each dialect/language.

Next, a closed-set dialect test is performed (i.e., the same train data for each dialect is tested against all dialect models in order to obtain score distributions from matched and mismatched train data models). Score distributions, derived from the score histograms are the basis for the proposed dialect separation measure. In order to compare two dialects \(D_1\) and \(D_2\), \(D_1\) training data is tested against \(D_1\) and \(D_2\) GMM models to obtain two sets of log likelihood scores: \(S_{11}\) and \(S_{12}\), respectively. \(S_{21}\) and \(S_{22}\) are obtained in a similar manner. Next, a histogram is formulated for each score set, and is approximated with a probability distribution function. We used the generalized extreme value (GEV) probability distribution (Kotz and Nadarajah 2000) to model the score histograms. The GEV distribution is a flexible three-parameter model that combines three types of extreme value issues into the distribution model.

The GEV has the following PDF with location parameter \(\mu \), scale parameter \(\sigma \), and shape parameter \(k\):where \(z = \frac{{x - \mu }}{\sigma }\), and \(\sigma > 0\). The range of definition of the distribution depends on the shape parameter, and for \(k \ne 0\): \(1 + kz > 0\). The parameters of the GEV score distribution are estimated based on maximum likelihood estimation.

The core idea here is that the more distinguishable the distributions are for the score sets \(S_{11}\) and \(S_{12}\), the greater the distance from \(D_1\) to \(D_2\) or \(d(D_1,D_2)\). Similarly, comparing the \(S_{21}\) and \(S_{22}\) score distributions yields the distance from \(D_2\) to \(D_1\) or \(d(D_2,D_1)\), which is not necessarily equal to \(d(D_1,D_2)\). Figure 2 shows \(S_{11}\) and \(S_{12}\) score distributions when \(D_1\) and \(D_2\) represent two Arabic dialects of AE (United Arab Emirates) and EG (Egypt), respectively. The underlying assumption is that the true spectral mismatch between two dialects can accurately be measured using MFCC features and GMMs.

Here, we employ the mean square error criterion to compare score distributions, and obtain an estimate of the proximity between each dialect pair. Our initial experiments showed that when two score distributions do not have considerable overlap, the corresponding dialects are far apart and can be well classified. However, the inverse is not always true. In other words, there are cases where the two distributions have measurable overlap, but the classification accuracies are still high. The reason is that when comparing score distributions, the score statistics of the entire data set are compared. Alternatively, in a classification task, only two scores at a time are compared which correspond to the same test stream. The solution to this problem is to move the analysis from a 2D surface to a 3D space.

In order to assess the separation between dialects \(D1\) and \(D2\), represented as \(d(D_1,D_2)\), an estimate is formed to determine how well \(D_1\) can be identified from \(D_2\). This estimation involves building a 3D score distribution based on two 2D score distributions via testing \(D_1\) data against the \(D_1\) and \(D_2\) models. Let us refer to these distribution functions as \(f_1\) and \(f_2\), respectively. The joint PDF of the two sets of scores is calculated as follows:This 3D distribution has the partial distributions of \(f_1\) and \(f_2\) in the XZ and YZ planes, respectively. Note that the \(S_{11}\) and \(S_{12}\) score distributions are not generally independent. Therefore, \(f\) is not exactly the distribution for the pairs of scores which result from testing each original train (now test) token in \(D_1\) against the \(D_1\) and \(D_2\) models. However, it does represent a good approximation that reflects the separation of the dialects. Next, the volume under \(f\) which lies between the YZ and the (X = Y)Z planes is calculated. This volume corresponds to the accumulated amount of dialect \(D_1\)’s correctly classified tokens for which the score against \(D_1\) model is higher than the outside dialect \(D_2\) model, and yields an estimation for the separation between \(D_1\) and \(D_2\):Here, a primary aim is to keep the dialect separation assessment process as simple as possible. Therefore, this procedure can serve as an initial step prior to the actual classification, in order to give the user an estimate of how effective the results of dialect classification might be. All score analysis in the proposed method is based on the training data and therefore represents a close-set test (for \(D_1\) against \(D_1\) GMM, but an open test when \(D_1\) data is tested against \(D_2\) GMM). Figure 3 shows the 3D score distribution for the 2D distributions depicted in Fig. 2. The figure shows the contour distribution of the log likelihood scores from pairs of Arabic dialects. Contours of equal likelihood are projected onto the XY plane in the lower portion, and a bi-secting plane is constructed to determine a decision surface to obtain a dialect separation volume measure.

To calculate \(\text {d}(D_2,D_1)\), \(S_{21}\) and \(S_{22}\) are applied in a similar manner. Next, \(\text {d}(D_1,D_2)\) and \(\text {d}(D_2,D_1)\) are averaged to obtain one combined proximity measure for the pair of dialects:The range of the proposed measure is [0.5,1]. The smallest distance between two dialects is 0.5, which is the worst case scenario for a dialect identification task. It occurs when the dialects are identical. However, if the data used in training the dialect models is open or separate from that used to assess the separation, the resulting distance may be slightly larger than 0.5, which is due to the variability within a dialect. The greatest distance is 1 which reflects two completely distinct and separated dialects. If the distance is less than 0.5, it implies that \(D_1\) data points are more likely to have the same dialect as \(D_2\). This can occur due to the impurity of the training data. Since one of the objectives of this study is assessment of the training data for dialects, if the distance is lower than 0.5, it means that the majority of \(D_1\) training data is in fact from \(D_2\) and the resulting model is not reliable for classification purposes.

Human perception tests indicate that prosodic cues can be employed to distinguish one language or accent from another (Muthusamy et al. 1994; Kumpf and King 1997). However, prosodic features have only briefly been considered for language identification (LID) systems (Zissman and Berkling 2001), and even less for dialect classification. Thyme-Gobbel and Hutchins (1996) explored the utility of syllable based parameters extracted from pitch and amplitude contours in LID tasks. Their results showed that prosodic cues alone render results comparable to many non-prosodic systems for some language pairs. Tong et al. (2006) integrated different levels of features for language identification, including prosodic features. Their study showed that different levels of discriminative features provide complementary cues for LID. In this section, we explore prosodic differences between dialects/languages, with the primary focus on pitch movement differences. Prosodic features including fundamental frequency patterns, have a suprasegmental nature (i.e., they cannot be associated with a single phone-sized segment) (Wightman and Ostendorf 1994). Therefore, modeling prosody is still an open ended problem (Rouas 2007). Syllable based speech units have been used previously for prosodic feature extraction. However, these approaches use segmentation as front-end processing. Manual segmentation of the speech signal can be costly in terms of time for large corpora. Alternatively, automatic segmentation methods introduce errors which can bias overall results. Rouas (2007) used pseudosyllables as the prosodic units, which are automatically extracted from input audio stream speech data. Adami et al. (2003) modeled pitch and energy contour dynamics for speaker recognition using linear piecewise stylization.

This study, concentrates on conversational data without any manual labeling or transcription. During conversations between speakers of the same dialect, speakers generally use more dialect specific language compared to directed read data. This has been observed for accent classification by Angkititrakul and Hansen (2006). The method employed here focuses on local variations of the pitch and energy contours which are compared among dialects. The proposed text-independent prosody features do not require segmentation.

A method for dialect separation assessment is proposed which compares statistical models of pitch contour details. An earlier version of this method was presented by Mehrabani et al. (2010). As a first step, a single pitch vector per utterance is obtained by first extracting pitch frequencies from every utterance of each dialect. The robust algorithm for pitch tracking (RAPT), proposed by Talkin (1995) is used for pitch extraction. RAPT is based on the normalized cross-correlation function (NCCF), and applies dynamic programming as a post-processing technique to select the best \(F_0\) and voicing state candidates at each frame. Next, 3-Dimensional feature vectors are generated from groups of three consecutive nonzero pitch values. To obtain a representation of the pitch contour microstructure, rather than speaker/utterance dependent absolute pitch values, pitch slopes are subsequently extracted from the 3-Dimensional pitch vectors. Since the step size in pitch extraction is fixed (10 ms), a feature directly proportional to pitch slope is calculated as the difference between consecutive pitch values, transforming the pitch vector [\(F_{0_1}\) \(F_{0_2}\) \(F_{0_3}\)] into a 2D vector [(\(F_{0_2}\)–\(F_{0_1}\)) (\(F_{0_3}\)–\(F_{0_2}\))]. For the remainder of this study, this extracted feature will be referred to as pitch slope. Figure 4 shows the example feature extraction from a pitch contour. As shown, an analysis window slides along the pitch contour, extracting three nonzero pitch values at a time. There is overlap of two samples between adjacent windows.

In the next step, the pattern of changes in every three consecutive pitch values is determined, using the 2D pitch slopes. A positive slope implies an increase in pitch, and alternatively, a negative slope represents a decrease. The absolute value of the slope or pitch change is also important. Pitch slopes close to zero, independent of the sign, represent almost flat fragments of the pitch contour. However, steep slopes correspond to abrupt changes in pitch.

In order to obtain a codebook of pitch patterns for 2D pitch slope vectors, a threshold is set for pitch slopes based on studying pitch slope histograms for all dialects. For each dialect, 2D pitch slope feature vectors extracted from every speaker and utterance of the dialect’s data are used to build a 3D histogram as the statistical representation of pitch change in that dialect. Figure 5 shows an example of a 3D pitch slope histogram. Each 2D pitch slope vector corresponds to a point on the \(XY\) plane.

A set of 9 distinct patterns are considered for each dialect, depicted in Fig. 6. If the absolute change of pitch is less than 3 \(Hz\), the pitch is considered unchanged. However, for absolute pitch slopes greater than 3 Hz, two options are considered: positive and negative.

After classifying all 2D pitch slope features as one of the 9 pitch patterns, the next step is to model pitch changes in each dialect. These models are later compared to obtain pitch movement differences between different dialects of a language. Statistical models are used here with discrete probability distributions. Each distribution shows the probability of occurrence for each pattern in the given dialect, and can be described by the probability matrix \(P\)(\(3 \times 3\)). The variability of these distributions reflects overall differences in the excitation structure between dialects.

Next, the obtained pitch pattern model profiles for 3D pitch vectors are exploited to build statistical models for longer temporal pitch patterns by means of N-gram modeling. The codebook of 9 pitch patterns from Fig. 6 is considered as a dictionary of different words: \(\big \{ w_1 ,w_2 ,w_3 ,w_4 ,w_5 ,w_6 ,w_7 ,w_8 ,w_9 \big \}\). The unigram counts for this dictionary are already calculated, which are the probabilities of occurrence for each word (pattern). Conditional probabilities are computed from the N-gram frequency counts:where \(C\) represents the count of the word sequences. Using the conditional probabilities, the probability of different sequences of pitch contour patterns can be calculated as,The same approach is also performed on energy contours to compare the statistics of log energy contour patterns among dialects, also using an entry codebook such as that in Fig. 6, with a slope threshold of 0.05 in place of the 3 Hz value used for pitch. The KL divergence is used to compare pitch/energy pattern models. If \(P\) and \(Q\) are two discrete probability distributions, the KL divergence of \(Q\) from \(P\) is:Having formulated the statistical models for pitch contour and energy contour distributions, along with their temporal language models in this section, combined with the statistical MFCC based method in the previous section, we now turn to an evaluation of dialect assessment in the next section.

Results from the proposed approaches for dialect proximity assessment are presented and compared for three Arabic dialects: United Arab Emirates (AE), Syria (SY), and Egypt (EG), as well as three South Indian (Dravidian) languages: Kannada (KAN), Tamil (TAM), and Telugu (TEL). Approximately 3 h of conversational speech from 32 male speakers for each dialect, and 6 h of conversational speech from 74 male speakers for each language was used as train data. Conversations were held between two speakers of the same dialect/language. Each speaker’s part of the conversation was recorded separately. The entire training data set was used for this evaluation.

Each assessment consists of three measurements corresponding to the three pairwise dialect/language comparisons. Dialect proximity assessment scores from different methods will naturally have their own numerical ranges. Therefore, the numerical scores in each set are normalized in order for an effective comparison. Figure 7 schematically shows three distances \({d_1} \ge {d_2} \ge {d_3}\) between pairs of a set of three dialects/languages, represented as \(D_1\), \(D_2\), \(D_3\). The normalization process consists of subtracting the minimum distance from each distance and dividing by the dynamic range of the distances. Next, each normalized distance is multiplied by 5 and added to 5 in order to map the largest distance (\(d_1\)) to 10, and the smallest distance (\(d_3\)) to 5:where \({d_i}^\prime \) represents the final normalized distance, with \(i=1,2,3\), using \(j=1,2,3\).

Figure 8 shows the normalized measures from the log likelihood score distribution and pitch pattern methods for the 3-way Arabic dialect set. For each method, a set of three distances are depicted as a triangle, which reflects three scores for the dialect pairs: (AE,SY), (AE,EG), (SY,EG). Each vertex represents a dialect, and length of each side is proportional to the distance between dialects represented at the vertices. Since the distances are normalized as explained, in each triangle the length of the largest side is 10 and the length of the smallest side is 5. As shown in the figure, for all three methods, the largest triangle side corresponds to distance between AE and SY, and the smallest side corresponds to distance between SY and EG. In other words, among these three Arabic dialects, AE and SY are the most separate pair, while SY and EG are the closest dialect pair. This means that the log likelihood, pitch pattern unigram, and pitch pattern bigram methods yield the same order of the proximity scores for this corpus. However, the results from pitch pattern unigram comparison are closer to the log likelihood method.

Figure 9 compares the normalized distances from the log likelihood score distribution, pitch pattern bigram, and energy pattern bigram methods for pairs of three South Indian languages. As shown in the figure, KAN and TAM have the largest distance among these language pairs with three methods. Next, repeatability and consistency of the proposed measures and the amount of data required for a reliable proximity assessment are evaluated for South Indian languages based on the increased amount of data for this corpus compared to the dialect database.

In this study, the goal of assessing dialect/language proximity in a 3-way set was considered. Intrinsic differences between dialects were studied, including spectral acoustic, as well as excitation structure differences. First, a method for measuring dialect separation was proposed based on a volume space analysis in a 3D model for GMM output score distributions. Next, prosody-based proximity measures were proposed, comparing statistical models for pitch/energy contour movement patterns between dialects. The proposed measures were evaluated on a corpus of Arabic dialects and a corpus of South Indian languages. The proposed dialect proximity assessment was shown to be consistent and repeatable. Future advances could consider lexical differences including word selection, grammar structure, or wider suprasegmental differences. Further high level linguistic knowledge concerning the evolution of specific dialects could also be considered for future dialect assessment strategies.