Bayesian estimation of Dirichlet mixture model with variational inference

Highlights

• An analytically tractable solution for Bayesian estimation of the Dirichlet mixture model.

• Relative convexity of the multivariate log-inverse-gamma function is proved and utilized.

• The free energy function is approximated by a single lower-bound to guarantee convergence.

• The method outperforms the ML based method and the VI based method, moreover, it is comparable to the sampling based method.

• The performances are demonstrated with important multimedia signal processing applications.

Abstract

In statistical modeling, parameter estimation is an essential and challengeable task. Estimation of the parameters in the Dirichlet mixture model (DMM) is analytically intractable, due to the integral expressions of the gamma function and its corresponding derivatives. We introduce a Bayesian estimation strategy to estimate the posterior distribution of the parameters in DMM. By assuming the gamma distribution as the prior to each parameter, we approximate both the prior and the posterior distribution of the parameters with a product of several mutually independent gamma distributions. The extended factorized approximation method is applied to introduce a single lower-bound to the variational objective function and an analytically tractable estimation solution is derived. Moreover, there is only one function that is maximized during iterations and, therefore, the convergence of the proposed algorithm is theoretically guaranteed. With synthesized data, the proposed method shows the advantages over the EM-based method and the previously proposed Bayesian estimation method. With two important multimedia signal processing applications, the good performance of the proposed Bayesian estimation method is demonstrated.

Introduction

Statistical modeling plays an important role in various research areas [1], [2], [3]. It provides a way to connect the data with the statistics. An essential part in statistical modeling is to estimate the values of the parameters in the distribution or to estimate the distribution of the parameters, if we consider them as random variables. The maximum likelihood (ML) estimation method gives point estimates to the parameters and disregards the remaining uncertainty in the estimation. Rather than taking the point estimates, the Bayesian estimation method gives the posterior probability distributions over all model parameters, using the observed data together with the prior distributions [3]. In general, compared to the ML estimation, the Bayesian estimation of the parameters in a statistical model could yield a robust and stable estimate, by including the resulting uncertainty into the estimation, especially when the amount of the observed data is small [4].

The Gaussian distribution and the corresponding Gaussian mixture model (GMM) are widely used to model the underlying distribution of the data. However, not all data we would like to model can be safely assumed to be Gaussian distributed [5]. Recently, the studies of non-Gaussian statistical models have become popular for the purpose of modeling bounded or semi-bounded data (see e.g., [6], [7], [8], [9]). The non-Gaussian statistical models include, among others, the beta distribution, the gamma distribution, and the Dirichlet distribution.

The Dirichlet distribution and the corresponding Dirichlet mixture model (DMM) were frequently applied to model proportional data, for example, in image processing [10], in text analysis [11], and in data mining [12]. For speech processing, applications of Dirichlet distribution in the line spectral frequency (LSF) parameter quantization [13] were shown superior to conventional GMM based methods. Another usage of the Dirichlet distribution is to model the probabilities of the weighting factors in a mixture model [14], [15]. In non-parametric Bayesian modeling, the Dirichlet process is actually an infinite-dimensional generalization of the Dirichlet distribution so that an infinite mixture model can be obtained [15], [16], [17]. Here, we study only the finite DMM and the work conducted can also be extended to the infinite mixture modeling case.

In this paper, we carry on our previous study of Bayesian analysis of BMM [8] and extend it to the Bayesian analysis of DMM. The parameters in a Dirichlet distribution are assumed mutually independent and each of them is assigned by a gamma prior. Although this assumption violates the correlation among the parameters, it captures the non-negative properties of those parameters. By this assumption, we can apply the factorized approximation (FA) method to carry out the Bayesian estimation. However, as the expectation of the multivariate log-inverse-beta (MLIB) function cannot be calculated explicitly, an analytically tractable solution to the posterior distribution is not feasible. To overcome this problem, we study some relative convex properties of the MLIB function. Using these convexities, we approximate the expectation of the MLIB function by a single lower-bound (SLB). With this derived SLB and by principles of the VI framework and the extended factorized approximation (EFA) method [8], [18], [19], [20], [21], [22], [23], [24], we approximate the posterior distributions of the parameters in a Dirichlet distribution with a product of several mutually independent gamma distributions, which satisfies the conjugate match between the prior and posterior distributions. Finally, an analytically tractable solution for calculating the posterior distribution is obtained. This analytically tractable solution avoids the numerical calculations in the EM algorithm [25], [10].

The proposed method, which is a full Bayesian framework, can automatically determine the model complexity (in terms of the number of necessary mixture components) based on the data. This task is also challenging in model estimation and the ML estimation itself cannot handle this issue. Moreover, the overfitting problem in the ML estimation can also be prevented due to the advantages of Occam׳s razor effect in Bayesian estimation. With synthesized data evaluation, the effectiveness and the accuracy of the proposed Bayesian estimation method over the ML estimation method [10], [25] and the recently proposed Bayesian estimation method [12] are demonstrated. For the real life applications, we evaluate the proposed Bayesian estimation method with two important multimedia signal processing applications, namely (1) the LSF parameter quantization in speech coding [13] and (2) the multiview depth image enhancement in free-viewpoint television (FTV) [26]. For both applications, the proposed Bayesian method works well and shows improvement over the conventional methods.

The remaining parts of this paper are organized as follows: the DMM and the Bayesian analysis of a DMM are introduced in Sections 2 and 3, respectively. In Section 4, we show the efficiency and good performance of the proposed method with the synthesized data and the real life data. Some conclusions are drawn in Section 5.