Joint DOA and multi-pitch estimation based on subspace techniques

Abstract

In this article, we present a novel method for high-resolution joint direction-of-arrivals (DOA) and multi-pitch estimation based on subspaces decomposed from a spatio-temporal data model. The resulting estimator is termed multi-channel harmonic MUSIC (MC-HMUSIC). It is capable of resolving sources under adverse conditions, unlike traditional methods, for example when multiple sources are impinging on the array from approximately the same angle or similar pitches. The effectiveness of the method is demonstrated on a simulated an-echoic array recordings with source signals from real recorded speech and clarinet. Furthermore, statistical evaluation with synthetic signals shows the increased robustness in DOA and fundamental frequency estimation, as compared with to a state-of-the-art reference method.

1. Introduction

The problem of estimating the fundamental frequency, or pitch, of a period waveform has been of interest to the signal processing community for many years. Fundamental frequency estimators are important for many practical applications such as automatic note transcription in music, audio and speech coding, classification of music, and speech analysis. Numerous algorithms have been proposed for both the single- and multi-pitch scenarios [1–5]. The problem for single-pitch scenarios is considered as well-posed. However, in real-world signals, the multi-pitch scenario occurs quite frequently [2, 6]. The multi-pitch estimation algorithms are often based on, i.e., various modification of the auto-correlation function [1, 7], maximum likelihood, optimal filtering, and subspace techniques [2, 3, 8]. In real-life recordings, problems such as frequency overlap of sources, reverberation, and colored noise will strongly limit the performance of multi-pitch estimator and estimator designed for single channel recordings often use simplified signal models. One widely used signal simplification in multi-pitch estimators, for example, is the sparseness of the signal, where the frequency spectrum of sources are assumed to not overlap [2]. This assumption may be appropriate when sources consist of mixture of several speech signals having different pitches [9]. However, for audio signals it is less likely to be true. This is especially so in western music, where instruments are most often played in accord, something that causes the harmonics to overlap or even coincide. With only single-channel recording it is, therefore, hard, or perhaps even impossible, to estimate pitches with overlapping harmonics, unless additional information, such as a temporal or spectral model, is included.

Recently, multi-channel approaches have attracted considerable attention both in single- and multi-pitch scenarios. By exploring the spatial information of the sources, more robust pitch estimators have been proposed [10–14]. Most of those multi-channel methods are still mainly based on auto-correlation function-related approaches, however, although a few exceptions can be found in [15–18]. In direction-of-arrival (DOA) estimators, audio and speech signals are often modeled as broadband signal, and standard subspace methods such as MUSIC and ESPRIT are only defined for narrow-band signal model, which then fail to directly operate on broadband signals [19]. One often used concept is band-pass filtering of broadband signals into subbands, where narrow-band estimators can be applied to each subband [20]. In the narrow-band case, a delay in the signal is equivalent to a phase shifts according to the frequencies of complex exponentials. An alternative method is, however, as follows: since harmonic signals consist of sinusoidal components, we can model each source as multiple narrow-band signal with distinct frequencies arriving at the same DOA.

In this article, we propose a parametric method for solving the problem of joint fundamental frequency and DOA estimation based on subspace techniques where the quantities of interest are jointly estimated using a MUSIC-like approach. We term the proposed estimator Multi-channel multi-pitch Harmonic MUSIC (MC-HMUSIC). The spatio-temporal data model used in MC-HMUSIC is based on the JAFE data model [21, 22]. Originally, the JAFE data model was used for estimating joint unconstrained frequencies and DOAs estimates of complex exponential using ESPRIT, which is referred as joint angle-frequency estimation (JAFE) algorithm. Other-related work with joint frequency-DOA methods includes [23–25]. In this article, we have parametrized the harmonic structure of periodic signals in the signal model to model the fundamental frequency and the DOA of individual sources. An estimator is constructed for jointly estimating the parameters of interest. Incorporating the DOA parameter in finding the fundamental frequency may give better robustness against a signal with overlapping harmonics. Similarly, it can be expected that the DOA can be found more accurately when the nature of the signal of interest is taken into account.

The remainder of this article is comprised four sections: Section 2, in which we will introduce some notation, the spatio-temporal signal model, for which we also derive the associated Cramér-Rao lower bound, along with the JAFE data mode; Section 3, where we then present the proposed method; Section 4, in which we present the experimental results obtained using the proposed method; and, finally, Section 5, where we conclude on our work.

2. Fundamentals

2.1. Spatio-temporal signal model

Next, the signal model employed throughout the article will be presented. Without multi-path propagation of sources, it is given as follows: the signal x _i received by microphone element i arranged in a uniform linear array (ULA) configuration, i = 1,..., M, is given by

$$\begin{array}{ll}\hfill x_i\left(n\right)& =\sum _{k=1}^K\sum _{l=1}^{L_k}{\beta }_{l,k}{\mathsf{\text{e}}}^{j\left({\omega }_{k}ln+{\varphi }_{k}l\left(i-1\right)\right)}+e_i\left(n\right),\\ \hfill {\beta }_{l,k}& =A_{l,k}{\mathsf{\text{e}}}^{j{\gamma }_{l,k}},\end{array}$$

(1)

for sample index n = 0,..., N - 1, where subscript k denotes the k th source and l the l th harmonic. Moreover, A _l,k is the real-valued positive amplitude of the complex exponential, L _k is the number of harmonics, K is number of sources, γ _l,k is the phase of the individual harmonics, ϕ _k is the phase shift caused by the DOA, and e _i (n) is complex symmetric white Gaussian noise. The phase shift between array elements is given as ${\varphi }_k={\omega }_k{f}_s\frac{d}{c}\text{sin}\left({\theta }_k\right)$, where d is the spacing between the elements measured in wavelengths, c is the speed of propagation in unit [m/s], θ _k is the DOA defined for θ _k ∈ [-90°, 90°], f _s is the signal sampling frequency. The problem of interest is to estimate ω _k and θ _k . We in the following assume that the number of sources K is known and the number of harmonics L _k of individual sources is known or found in some other, possibly joint, way. We note that a number of ways of doing this has been proposed in the past [26–28, 2].

2.2. Cramér-Rao lower bound

We will now proceed to derive the exact Cramér-Rao lower bound (CRLB) for the problem of estimating the parameters of interest. First, we define the M × 1 deterministic signal model vector s(n, μ) with column element as

$$s_i\left(n,\mu \right)=\sum _{k=1}^K\sum _{l=1}^{L_k}{\beta }_{l,k}{\mathsf{\text{e}}}^{j\left({\omega }_{k}ln+{\varphi }_{k}l\left(i-1\right)\right)},{\beta }_{l,k}=A_{l,k}{\mathsf{\text{e}}}^{j{\gamma }_{l,k}},$$

(2)

where s(n, μ) = [s₁(n, μ) ... s _M (n, μ)]^T. Furthermore, the parameter vector μ is given by

$$\begin{array}{c}\mu =[{\omega }_1\cdot \cdot \cdot {\omega }_K{\theta }_1\cdot \cdot \cdot {\theta }_K\\ A_{1,1}{\gamma }_{1,1}\cdot \cdot \cdot A_{L_K,K}{\gamma }_{L_K,K}].\end{array}$$

(3)

Recall that the observed signal vector with additive white noise is given by

$$\mathbf{x}\left(n\right)=\mathbf{s}\left(n,\mu \right)+\mathbf{e}\left(n\right)=\left[\begin{array}{c}\hfill s_1\left(n,\mu \right)\hfill \\ \hfill ⋮\hfill \\ \hfill s_M\left(n,\mu \right)\hfill \end{array}\right]+\mathbf{e}\left(n\right),$$

(4)

with e(n) being the noise column vector. The CRLB is defined as the variance of an unbiased estimate of the p th element of μ, which is lower bounded as

$$\mathsf{\text{var}}\left({\mu }_p\right)\ge {\left[{\mathbf{C}}^{-1}\right]}_{pp},$$

(5)

where C is the so-called Fisher information matrix given by

$$\mathbf{C}=\frac{2}{{\sigma }^2}\mathsf{\text{Re}}\left(\sum _{n=0}^{N-1}\frac{\partial \mathbf{s}{\left(n,\mu \right)}^H}{\partial \mu }\frac{\partial \mathbf{s}\left(n,\mu \right)}{\partial {\mu }^T}\right).$$

(6)

The partial derivative matrix is denoted as

$$\frac{\partial \mathbf{s}\left(n,\mu \right)}{\partial \mu }=\left[\frac{\partial s_1\left(n,\mu \right)}{\partial \mu }\cdot \cdot \cdot \frac{\partial s_M\left(n,\mu \right)}{\partial \mu }\right],$$

(7)

where vector $\frac{\partial s_i\left(n,\mu \right)}{\partial \mu }$ is the partial derivatives with respect to the entries in the vector μ. The expression for the columns in $\frac{\partial \mathbf{s}\left(n,\mu \right)}{\partial \mu }$ is given as

$$\frac{\partial s_i\left(n,\mu \right)}{\partial \mu }=\left[\begin{array}{c}\sum _{l=1}^{L_1}jl\left(n+\left(i-1\right)f_s\frac{d}{c}\text{sin}\left({\theta }_1\right)\right){\beta }_{l,1}{\mathsf{\text{e}}}^{j\left({\omega }_{1}ln+{\varphi }_{1}l\left(i-1\right)\right)}\\ ⋮\\ \sum _{l=1}^{L_K}jl\left(n+\left(i-1\right)f_s\frac{d}{c}\text{sin}\left({\theta }_K\right)\right){\beta }_{l,K}{\mathsf{\text{e}}}^{j\left({\omega }_{K}ln+{\varphi }_{K}l\left(i-1\right)\right)}\\ \sum _{l=1}^{L_1}\left(jl\left(i-1\right){\omega }_1{f}_s\frac{d}{c}\text{cos}\left({\theta }_1\right)\right){\beta }_{l,1}{\mathsf{\text{e}}}^{j\left({\omega }_{1}ln+{\varphi }_{1}l\left(i-1\right)\right)}\\ ⋮\\ \sum _{l=1}^{L_K}\left(jl\left(i-1\right){\omega }_K{f}_s\frac{d}{c}\text{cos}\left({\theta }_K\right)\right){\beta }_{l,K}{\mathsf{\text{e}}}^{j\left({\omega }_{K}ln+{\varphi }_{K}l\left(i-1\right)\right)}\\ {\mathsf{\text{e}}}^{j{\gamma }_{1,1}}{\mathsf{\text{e}}}^{j\left({\omega }_{1}n+{\varphi }_1\left(i-1\right)\right)}\\ j{A}_{1,1}{\mathsf{\text{e}}}^{j{\gamma }_{1,1}}{\mathsf{\text{e}}}^{j\left({\omega }_{1}n+{\varphi }_1\left(i-1\right)\right)}\\ ⋮\\ {\mathsf{\text{e}}}^{j{\gamma }_{L_K,K}}{\mathsf{\text{e}}}^{j\left({\omega }_K{L}_{K}n+{\varphi }_K{L}_K\left(i-1\right)\right)}\\ j{A}_{L_K,K}{\mathsf{\text{e}}}^{j{\gamma }_{L_K,K}}{\mathsf{\text{e}}}^{j\left({\omega }_K{L}_{K}n+{\varphi }_K{L}_K\left(i-1\right)\right)}\end{array}\right].$$

(8)

2.3. The JAFE data model

Next, we will introduce the specifics of the JAFE data model [22, 29] that our method is based on. At a time instant n the received signal from the M array elements are x(n) = [x₁(n) x₂(n) ... x _M (n)]^T, which can be written as

$$\mathbf{x}\left(n\right)=\mathbf{A}\Phi \mathbf{b}+\mathbf{e}\left(n\right),$$

(9)

where e(n) ∈ ℂ^M×1 is the noise vector, and A = [A₁ ... A _K ] is a Vandermonde matrix containing parameters ω _k and θ _k for sources k = 1, . . . , K, i.e.,

$${\mathbf{A}}_k=\left[\mathbf{a}\left({\theta }_k,{\omega }_{k}1\right)\cdot \cdot \cdot \mathbf{a}\left({\theta }_k,{\omega }_k{L}_k\right)\right],$$

(10)

with a(θ, ω) being the array steering vector given by

$$\mathbf{a}\left(\theta ,\omega \right)={\left[\mathsf{\text{1}}\cdot \cdot \cdot {\mathsf{\text{e}}}^{j\omega f_s\frac{d}{c}\left(M-1\right)\text{sin}\left(\theta \right)}\right]}^T.$$

(11)

Here, (·)^T denotes the vector transpose. Unlike the steering vector defined in [22, 21], where only the DOA is parametrized, here, a general definition of the vector (11) is used, in which it depends on both θ and ω [29]. The frequency components are expressed in ${\Phi }^n=\mathsf{\text{diag}}\left(\left[{\Phi }_1^n\cdots {\Phi }_K^n\right]\right)$where the matrix for each source is given by

$${\Phi }_k=\mathsf{\text{diag}}\left(\left[{\mathsf{\text{e}}}^{j{\omega }_k}\cdot \cdot \cdot {\mathsf{\text{e}}}^{j{\omega }_k{L}_k}\right]\right).$$

(12)

The complex amplitudes for involving components are represented by the following vector:

$$\mathbf{b}={\left[{\beta }_{1,1}\cdot \cdot \cdot {\beta }_{L_1,1}\cdot \cdot \cdot {\beta }_{1,K}\cdot \cdot \cdot {\beta }_{L_K,K}\right]}^T.$$

(13)

To capture the temporal behavior, N time-domain data samples of the array output x(n) are collected to form the M × N data matrix X, which is defined as

$$\mathbf{X}=\left[\mathbf{x}\left(n\right)\cdot \cdot \cdot \mathbf{x}\left(N\right)\right].$$

(14)

Due to the structure of the harmonic components, the data matrix is given by

$$\mathbf{X}=\mathbf{A}\left[\mathbf{b}\Phi \mathbf{b}\cdot \cdot \cdot {\Phi }^{N-1}\mathbf{b}\right]+\mathbf{E},$$

(15)

where E ∈ ℂ^M×N is a matrix containing N sample of the noise vector e(n).

In speech and audio signal processing, it is common to model each source as a set of multiple harmonics with model order L _k > 1. Due to the narrow-band approximation of the steering vector, the multiple complex components with distinct frequencies impinge on the array with identical DOA will result in a non-unique spatial frequencies which cause a harmonic structure in the spatial frequencies ϕ _k l ∀l as well. The multiple sources impinge on the array with different DOAs consisting of various frequency components may, for certain frequency combinations, give the same array steering vector, which cause the matrix A to be rank deficient. Normally, this ambiguous mapping of the steering vector is mitigated by band-pass filtering the signal into its subbands, where the DOA of the signal is uniquely modeled by the narrow-band steering vector [20, Chap. 9].

Here, the ambiguities and the rank-deficiency are avoided by introducing temporal smoothness in order to restore the rank of A. The temporally smoothed data matrix is obtained by stacking t times temporally shifted versions of the original data matrix [22, 21, 29], given as

$${\mathbf{X}}_t=\left[\begin{array}{c}\mathbf{A}\left[\mathbf{b}\Phi \mathbf{b}\cdot \cdot \cdot {\Phi }^{N-t}\mathbf{b}\right]\\ \mathbf{A}\Phi \left[\mathbf{b}\Phi \mathbf{b}\cdot \cdot \cdot {\Phi }^{N-t}\mathbf{b}\right]\\ ⋮\\ \mathbf{A}{\Phi }^{t-1}\left[\mathbf{b}\Phi \mathbf{b}\cdot \cdot \cdot {\Phi }^{N-t}\mathbf{b}\right]\end{array}\right]+{\mathbf{E}}_t,$$

(16)

where X _t ∈ ℂ^tM×N-t+1 is the temporally smoothed data matrix, and E _t is the noise term constructed from E in a similar way as X _t . In using the signal model where the amplitudes are assumed stationary for n = 0, . . . , N - 1, X _t can be factorized as

$${\mathbf{X}}_t=\left[\begin{array}{c}\hfill \mathbf{A}\hfill \\ \hfill A\Phi \hfill \\ \hfill ⋮\hfill \\ \hfill \mathbf{A}{\Phi }^{t-1}\hfill \end{array}\right]\left[\mathbf{b}\Phi \mathbf{b}\cdot \cdot \cdot {\Phi }^{N-t}\mathbf{b}\right]+{\mathbf{E}}_t.$$

(17)

With some additional definitions, we can also write this expression more compactly as

$${\mathbf{X}}_t={\bar{\mathbf{A}}}_t{\mathbf{B}}_t+{\mathbf{E}}_t,$$

(18)

where Ā _t = [A AΦ ... AΦ^t-1]^T and B _t = [b Φb ... Φ^N-t b]. The temporally smoothed data matrix X _t can maximally resole up to $tM\ge {\sum }_{k=1}^K{L}_k$ complex exponentials, where Ā _t is linearly independent for any distinct θ and ω [30].

When multiple sources with distinct DOA with the same fundamental frequency impinge on the array, it will result in correlation between the underlying signals, which will make it harder to separate the corresponding components into its eigenvectors [22, 31]. To mitigate this problem, spatial smoothing is introduced, which works as follows. An array of M sensors is subdivided into S subarrays. In this article, the subarrays are spatially shifted with one element in each subarrays, the number of elements in each subarray being M _S = M - S + 1. For s = 1, . . . , S, let ${\mathbf{J}}_s\in {ℂ}^{t{M}_s\times tM}$ be the selection matrix corresponding to the s th subarray for the data matrix X _t . Then, the spatio-temporally smoothed data matrix ${\mathbf{X}}_{t,S}\in {ℂ}^{t{M}_s\times S\left(N-t+1\right)}$ is given by

$${\mathbf{X}}_{t,s}=\left[{\mathbf{J}}_1{\mathbf{X}}_t\cdot \cdot \cdot {\mathbf{J}}_S{\mathbf{X}}_t\right].$$

(19)

Furthermore, X _t,s can be factorized as

$${\mathbf{X}}_{t,s}=\left[{\mathbf{J}}_1{\bar{\mathbf{A}}}_t\cdot \cdot \cdot {\mathbf{J}}_S{\bar{\mathbf{A}}}_t\right]\left[\begin{array}{c}\hfill {\mathbf{B}}_t\hfill \\ \hfill \ddots \hfill \\ \hfill {\mathbf{B}}_t\hfill \end{array}\right]+{\mathbf{E}}_{t,s},$$

(20)

where E _t,s is the noise term constructed from E in a similar way as X _t,s . Using the shift invariance structure in A _m , the term J _s A _m for s = 1, . . . , S is given by

$${\mathbf{J}}_s{\bar{\mathbf{A}}}_t={\mathbf{J}}_1{\bar{\mathbf{A}}}_t{\Theta }^{s-1},$$

(21)

where

$$\Theta =\mathsf{\text{diag}}\left\{\left[{\mathsf{\text{e}}}^{j{\varphi }_{1}1}\cdots {\mathsf{\text{e}}}^{j{\varphi }_1{L}_1}\cdot \cdot \cdot {\mathsf{\text{e}}}^{j{\varphi }_{K}1}\cdots {\mathsf{\text{e}}}^{j{\varphi }_K{L}_K}\right]\right\},$$

(22)

which is simply the phase difference between array elements. With (21), the matrix X _t,s can be written in a compact form as

$${\mathbf{X}}_{t,s}={\mathbf{J}}_1{\bar{\mathbf{A}}}_t\left[{\mathbf{B}}_t\Theta {\mathbf{B}}_t\cdot \cdot \cdot {\Theta }^{S-1}{\mathbf{B}}_t\right]+{\mathbf{E}}_{t,s},$$

(23)

with selection matrix expressed as

$${\mathbf{J}}_1={\mathbf{I}}_t\otimes \left[{\mathbf{I}}_{M_s}\mathbf{0}\right],$$

(24)

where I _t ∈ ℝ^t×t and ${\mathbf{I}}_{M_s}\in {ℝ}^{M_s\times M_s}$ are the identity matrices, ⊗ is the Kroneker product as defined in [22].

It is interesting to note that the noise term E _t,s is no longer white due to the spatio-temporal smoothing procedure, as correlation between the different rows of (23) is obtained. A pre-whitening step can be implemented in (23) to mitigate this. We note, however, that according to results reported in [22], pre-whitening step is only interesting for signals with low SNR where minor estimation improvement can be achieved. In this article, the main interest is to propose a multi-channel joint DOA and multi-pitch estimator, for which reason the whitening process is left without further description, but we refer the interested reader to [22]. We also note that aside from spatial smoothing, forward-backward averaging could also be implemented to reduce the influence of the correlated sources [22, 31, 19].

3. The proposed method

3.1. Coarse estimates

From the final spatio-temporally smoothed data matrix, a basis for the signal and noise subspaces can be obtained as follows. The singular value decomposition (SVD) of the data matrix (23) is given by

$${\mathbf{X}}_{t,s}=\mathbf{U}\Sigma {\mathbf{V}}^H,$$

(25)

where the columns of U are the singular vectors, i.e.,

$$\mathbf{U}=\left[{\mathbf{u}}_1\cdot \cdot \cdot {\mathbf{u}}_{t{M}_{\mathsf{\text{S}}}}\right].$$

(26)

A basis of the orthogonal complement of the signal subspace, also called the noise subspace, is formed from singular vector associated with the mM _S - Q least significant singular values, i.e.,

$$\mathbf{G}=\left[{\mathbf{u}}_{Q+1}\cdot \cdot \cdot {\mathbf{u}}_{m{M}_{\mathsf{\text{S}}}}\right],$$

(27)

with $Q={\sum }_{k=1}^K{L}_k$being the total number of complex exponentials in the signal. Similarly, the signal subspace is spanned by the Q largest singular values, i.e.,

$$\mathbf{S}=\left[{\mathbf{u}}_1\cdot \cdot \cdot {\mathbf{u}}_Q\right].$$

(28)

The defined signal subspace and noise subspace have similar property as traditional subspaces where estimators such as joint DOA and frequency, or fundamental frequency estimators can be constructed using the principle used in MUSIC [19, 32, 27, 26, 4]. According to the signal noise subspace orthogonality principle, the following relationship holds:

$${\mathbf{J}}_1{\bar{\mathbf{A}}}_t\mathbf{G}=\mathbf{0},$$

(29)

where we, for notational simplicity, have introduced J₁Ā _t = A _ts . The matrix A _ts is comprised Vandermonde matrices for sources k = 1, . . . , K. The matrix for each individual source is given by

$${\mathbf{A}}_{ts,k}=\left[\begin{array}{ccc}\hfill 1\hfill & \hfill \cdots \hfill & \hfill 1\hfill \\ \hfill {\mathsf{\text{e}}}^{j{\varphi }_k}\hfill & \hfill {\mathsf{\text{e}}}^{j{\varphi }_k{L}_k}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill {\mathsf{\text{e}}}^{j{\varphi }_{k}S}\hfill & \hfill \cdots \hfill & \hfill {\mathsf{\text{e}}}^{j{\varphi }_k{L}_{k}S}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill {\mathsf{\text{e}}}^{j{\omega }_k\left(t-1\right)}\hfill & \hfill \cdots \hfill & \hfill {\mathsf{\text{e}}}^{j{\omega }_k{L}_k\left(t-1\right)}\hfill \\ \hfill {\mathsf{\text{e}}}^{j{\varphi }_k}{\mathsf{\text{e}}}^{j{\omega }_k\left(t-1\right)}\hfill & \hfill {\mathsf{\text{e}}}^{j{\varphi }_k{L}_k}{\mathsf{\text{e}}}^{j{\omega }_k{L}_k\left(t-1\right)}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill {\mathsf{\text{e}}}^{j{\varphi }_{k}S}{\mathsf{\text{e}}}^{j{\omega }_k\left(t-1\right)}\hfill & \hfill \cdots \hfill & \hfill {\mathsf{\text{e}}}^{j{\varphi }_k{L}_{k}S}{\mathsf{\text{e}}}^{j{\omega }_k{L}_k\left(t-1\right)}\hfill \end{array}\right].$$

(30)

The cost function of the proposed joint DOA and multi-pitch estimator is then

$$J\left({\omega }_k,{\theta }_k\right)={∥{\mathbf{A}}_{ts,k}^H\mathbf{G}∥}_F^2,$$

(31)

where ||·|| _F is the Frobenius Norm. Note that this measure is closely related to the angles between the subspaces as explained in [33] and can hence be used as a measure of the extent to which (29) holds for a candidate fundamental frequency and DOA. The pair of fundamental frequency and DOA can, therefore, be found as the combination that is the closest to being orthogonal to G, i.e.,

$${\left\{{\omega }_k,{\theta }_k\right\}}_{k=1}^K=\text{arg}\underset{{\left\{{\theta }_k\right\}}_1^K,{\left\{{\omega }_k\right\}}_1^K}{\text{min}}{∥{\mathbf{A}}_{ts,k}^H\mathbf{G}∥}_F^2.$$

(32)

The multi-channel estimators will have a cost function which is more well-behaved compared to those of single channel multi-pitch estimators (see, e.g., [26, 32, 28] for some examples of such).

3.2. Refined estimates

For many applications, only a coarse estimate of involved fundamental frequencies and DOAs are needed, in which case the cost function in (32) is evaluated on pre-defined search region with some specified granularity. If, however, very accurate estimates are desired, a refined estimate can be found as described next. For a rough estimate of the parameter of interests, refined estimates are obtained by minimizing the cost function in (32) using a cyclic minimization approach. The gradient of the cost function (32) for fundamental frequency and DOA are given as

$$\frac{\partial }{\partial {\omega }_k}J\left({\omega }_k,{\theta }_k\right)=2\mathsf{\text{Re}}\left(\mathsf{\text{Tr}}\left\{{\mathbf{A}}_{ts,k}^H\mathbf{G}{\mathbf{G}}^H\frac{\partial }{\partial {\omega }_k}{\mathbf{A}}_{ts,k}\right\}\right),$$

(33)

$$\frac{\partial }{\partial {\theta }_k}J\left({\omega }_k,{\theta }_k\right)=2\mathsf{\text{Re}}\left(\mathsf{\text{Tr}}\left\{{\mathbf{A}}_{ts,k}^H\mathbf{G}{\mathbf{G}}^H\frac{\partial }{\partial {\theta }_k}{\mathbf{A}}_{ts,k}\right\}\right),$$

(34)

with Re (·) denoting the real value. The gradient can be used for finding refined estimate using standard methods.

Here, we iteratively find a refined estimate using a cyclic approach. During an iteration, ω _k is first estimated with

$${\widehat{\omega }}_k^{i+1}={\widehat{\omega }}_k^i-\delta \frac{\partial }{\partial {\omega }_k}J\left({\widehat{\omega }}_k^i,{\widehat{\theta }}_k^i\right),$$

(35)

where i is the iteration index and δ is a small positive constant that is found using line search. The estimated ${\widehat{\omega }}_k^{i+1}$ is then used to initialize the minimization function for DOA, which is then found as

$${\widehat{\theta }}_k^{i+1}={\widehat{\theta }}_k^i-\delta \frac{\partial }{\partial {\theta }_k}J\left({\widehat{\omega }}_k^{i+1},{\widehat{\theta }}_k^i\right).$$

(36)

The method is initialized for i = 0 using the coarse estimates obtained from (32).

4. Experimental results

4.1. Signal examples

We start the experimental part of this article by illustrating the application of the proposed method to analyzing a mixed signal consisting of speech and clarinet signals, sampled at f _s = 8000 Hz. The single-channel signals are converted into a multi-channel signal by introducing different delays according to two pre-determined DOA to simulate a microphone array with M = 8 channels. The simulated DOAs of the speech and the clarinet signals are, respectively, θ₁ = -45° and θ₂ = 45°. The spectrogram of the mixed signal of the first channel is illustrated in Figure 1. To avoid spatial ambiguities, the distance between two sensor is half the wavelength of the highest frequency in the observed signal, here d = 0.0425 m. The mixed signal is segmented into 50% overlapped signal segments with N = 128. The user parameter selected in this experiment is $t=⌊\frac{2N}{3}⌋$ and $s=⌊\frac{M}{2}⌋$. The cost function is evaluated with a Vandermonde matrix with L = 5 complex exponentials, and the noise subspace is formed from an overestimated signal subspace with assumption of signal subspace containing N/2 = 64 complex exponentials. The signal subspace overestimation technique is usually used when the true order of the signal subspace is unknown, the signal subspace is assumed to be larger than the true one which can minimize the signal subspace components in the noise subspace. An added benefit of posing the problem as a joint estimation problem is that the multi-pitch estimation problem can be seen as several single-pitch problems for a distinct set of DOAs, one per source. Therefore, it is less important to select an exact signal model order than single-channel multi-pitch estimators would need [28]. The cost function is evaluated for frequencies from 100 to 500 with granularity of 0.52 Hz. The evaluated results are illustrated in Figure 2 where the upper panel contains the fundamental frequency estimates and lower panel the DOA estimates. It can be seen that the proposed algorithm can track the fundamental frequency and the DOA of the speech signal well, with only a few observed errors on regions with low signal energy. The clarinet signal's DOA and fundamental frequencies have also been estimated well for all segments.

Figure 1

The mixed spectrogram of the real recorded speech and clarinet signal.

Figure 2

The estimation results using the proposed methods: (a) fundamental frequency, (b) the DOA with the horizontal axis denoting time axis.

For the purpose of further comparison, the same signal will be analyzed using a standard time delay-and-sum beamformer [34] for DOA estimates and a single-channel maximum-likelihood based pitch estimator applied on the beamformed output signals [2]. The results are shown in Figure 3. The figure clearly shows that the delay-sum beamformer cannot satisfactory resolve the DOAs with M = 8 array elements which will further affect the performance of the single-channel pitch estimator, as shown in the upper panel. In this example, the proposed algorithm shown in Figure 2 is superior compared to reference method shown in Figure 3. The low resolution performance of the reference method will make the statistical evaluation of this method uninteresting, and we, therefore, will not be using it any further in the experiments to follow.

Figure 3

The estimates of (a) the fundamental frequency using maximum-likelihood estimator at the output of the beamformer, (b) the DOA using a delay-sum beamformer.

4.2. Statistical evaluation

Next, we use Monte Carlo simulations evaluated on synthetic signals embedded in noise in assessing the statistical properties of the proposed method and compare it with the exact CRLB. As a reference method for pitch and DOA estimation, we use the JAFE algorithm proposed in [22] for jointly estimating unconstrained frequencies and DOAs. Next, the unconstrained frequencies are grouped according to their corresponding DOAs where closely related directions are grouped together. A fundamental frequency is formed from these grouped frequencies in a weighted way as proposed in [35]. We refer this as the WLS estimator. In order to remove the errors due to the erroneous estimate of amplitudes, we assume WLS having the exact signal amplitude given. The WLS estimator is a computationally efficient pitch estimation method with good statistical properties. The reference DOA estimate is easily obtained in a similar way from the mean value of these grouped DOAs according to [22].

Here, we consider a M = 8 element ULA with sensor distance d = 0.0425 with a sampling frequency of f _s = 8000. The estimators are evaluated for two signal setups, first with two sources having ω₁ = 252.123 and ω₂ = 300.321 with L_1,2 = 3, and second with one harmonic source of ω₁ = 252.123 and L₁ = 3. All amplitudes on individual harmonics are set to unity A _k,l = 1 for tractability. Both sources are assumed to be far-field sources impinging on the array with DOAs at θ₁ = -43.23° and θ₂ = 70°, respectively, and for one source having a DOA of θ₁ = -43.23°. All simulation results are based on 100 Monte Carlo runs. The performance is measured using the root mean squared estimation error (RMSE) as defined in [28, 32, 26, 27]. The user parameter for JAFE data model is selected to the optimal values as proposed in [22] with temporal and spatial smoothness parameters, $t=⌊\frac{2N}{3}⌋$ and $s=⌊\frac{M}{2}⌋$, respectively. We note that in practical applications, the computational complexity has to also be considered in selecting the appropriate parameters t and s. An example of the 2-dimensional (2D) cost function of our proposed method evaluated on two mixed signal is illustrated in Figure 4, where a coarser estimate of the DOA and fundamental estimates can be identified from the two peaks in the 2D cost function.

Figure 4

Example of cost functions for two synthetic sources having three harmonic each, N = 64 and M = 8. The true fundamental frequency of ω₁ = 252.123 and ω₂ = 300.321 having DOA θ₁ = -43.23° and θ₂ = 70°, respectively.

In the first simulation, we evaluate the proposed method's statistical properties in a single source scenario for varying sample lengths and SNRs. The RMSEs on signal with varying N are shown in Figure 5, and with varying SNR in Figure 6. It can be seen from these figures that both estimators perform well for all SNR above 0 dB with WLS being slightly better for fundamental frequency estimation while the proposed estimator is better in DOA estimation. Both methods are also able to follow CRLB closely for around sample length N > 60. The better DOA estimation capabilities of the proposed method can be explained by the joint estimation of the fundamental frequency and DOA, which leads to increased robustness under adverse conditions. Both estimators can be considered as consistent in the single-pitch scenario.

Figure 5

RMSE as a function of N for SNR = 40 dB evaluated on single-pitch signal with unit amplitude: (a) fundamental frequency estimates; (b) DOA estimates.

Figure 6

RMSE as a function of SNR for N = 64 evaluated on single-pitch signal with unit amplitudes: (a) fundamental frequency estimates; (b) DOA estimates.

Next, we evaluate our method for the multi-pitch scenario. The so-obtained RMSEs for varying N and SNR are depicted in Figures 7 and 8. In Figure 7, it clearly shows that the proposed method is better than the WLS estimator for short sample lengths. The WLS estimator is not following CRLB until N > 80 samples while the proposed estimator is for N > 64. The remaining gap between CRLB and both evaluated estimators for N > 80 are due to the mutual interference between the harmonic sources. The slowly converging performance of WLS is mainly due to the bad estimate of the unconstrained frequency estimate using the JAFE method. With our selected simulation setup, the JAFE estimator is not giving consistent estimates for all harmonic components, which, in turn, results in poor performance in the WLS estimates. In general, the WLS estimator is sensitive to spurious estimate of the unconstrained frequencies. Moreover, the proposed estimator, which is jointly estimating both the DOA and the fundamental frequency, yields better estimates for smaller sample length N. The results in terms of RMSEs for varying SNRs are shown in Figure 8. This figure shows that the proposed estimator is again more robust than the WLS estimator for both DOA and fundamental frequency estimation.

Figure 7

RMSE as a function of N for SNR = 40 dB evaluated on multi-pitch signal with unit amplitudes: (a) joint fundamental frequency estimates; (b) joint DOA estimates.

Figure 8

RMSE as a function of SNR for N = 64 evaluated on multi-pitch signal with unit amplitudes: (a) joint fundamental frequency estimates; (b) joint DOA estimates.

In next two experiments, we will study the performance as a function of the difference in fundamental frequencies and DOAs for multiple sources. We start with studying the RMSE as a function of the difference between the fundamental frequencies of two harmonic sources, i.e., Δω = |ω₁ - ω₂|, with θ₁ = -43.321° and θ₂ = 70°. Here, we use an SNR set to 40 dB, and a sample length N = 64 with M = 8 array elements. The obtained RMSEs are shown in Figure 9. The figure clearly shows that both methods can successfully estimate the fundamental frequencies and DOAs. Once again the proposed estimator gives more robust estimates, close to the CRLB. Additionally, it should be noted that both methods are correctly estimating the DOA even when the both fundamental frequencies are identical ω₁ = ω₂, something that would not be possible with only a single channel. MC-HMUSIC has the ability to estimate the fundamental frequencies when both harmonics are identical provided that the DOAs are distinct and vice versa. Estimation of the parameters of signals with overlapping harmonics is a crucial limitation in multi-pitch estimation using only single-channel recordings. In the final experiment, the RMSE as a function of the difference between the DOAs of two harmonic sources Δθ = |θ₁ - θ₂| is analyzed for an SNR set to 40 dB and a sample length of N = 64 with M = 8 array elements. The fundamental frequencies are ω₁ = 252.123 and ω₂ = 300.321, respectively. The observations and conclusions are basically the same as before, with the proposed method outperforming the reference method so far.

Figure 9

RMSE as a function of Δ ω: (a) joint fundamental frequency estimates; (b) joint DOA estimates.

5. Conclusion

In this article, we have generalized the single-channel multi-pitch problem into a multi-channel multi-pitch estimation problem. To solve this new problem, we propose an estimator for joint estimation of fundamental frequencies and DOAs of multiple sources. The proposed estimator is based on subspace analysis using a time-space data model. The method is shown to have potential in applications to real signals with simulated anechoic array recording, and a statistical evaluation demonstrates its robustness in DOA and fundamental frequency estimation as compared to a state-of-the-art reference method. Furthermore, the proposed method is shown to have good statistical performance under adverse conditions, for example for sources with similar DOA or fundamental frequency.