A Music Cognition–Guided Framework for Multi-pitch Estimation

The objective of this work is to detect the multiple pitches from music pieces of mixed instruments, where an MPE system is proposed, which contains three key modules, i.e., pre-processing, harmonic structure detection and note tracking. Preprocessing covers a standard procedure, in which an input music signal needs to go through time–frequency (TF) representation and matrix factorization for feature extraction. The overall diagram of the MPE framework is illustrated in Fig. 1, where the implementation details are presented as follows.

According to the cognitive neuroscience of music [19, 28], before selectively stimulating the auditory cortex, different frequencies within the music need to be first filtered by human cochlea. As the frequency of human auditory perception is logarithmically distributed [27], there is a greater discrimination when hearing relatively lower frequencies. The Constant Q transform (CQT) [29], based on the FFT principle, can process a logarithmic compression similar to that of human’s cochlea helical structure [29]. Therefore, the CQT is employed as the TF representation module to derive the TF spectrogram, as it is efficient in lower frequencies. There are fewer frequencies required in a given range, which has testified its usefulness when the frequency distribution in several octaves is discrete. Meanwhile, an increased frequency bin correlates to a decrease in the temporal resolution rate, making it suitable for auditory applications. A spectral resolution of 60 bins per octave is used as suggested by Brown [27]. The outputs from the TF transformation are linear when using the Fast Fourier Transform (FFT) to analyse the frequency (Fig. 2a).

In the matrix factorization module, the CQT spectrogram results are used as the input, approximately modelled as a bivariate probability distribution \({\varvec{P}}\left(p,t\right)\). The output of this module is a 2-dimensional non-binary representation of pianoroll transcription (a pitch vs. time matrix shown in Fig. 2b). In this paper, the fast shift-invariant probabilistic latent component analysis (SI-PLCA) [30] is used for automatic transcription of polyphonic music, as it is extremely useful for log-frequency spectrogram, due to the same inter-harmonic spacing for all periodic sounds [31]. Given an input signal \({{\varvec{X}}}_{t}\), the output of CQT is a log-frequency spectrogram \({{\varvec{V}}}_{z,t}\) that can be considered as a joint time–frequency distribution \({\varvec{P}}\left(z,t\right)\) where \(z\) and \(t\) denote the frequency and time, respectively. After applying the SI-PLCA, \({\varvec{P}}\left(z,t\right)\) can be further decomposed into several components by [30]:where \(p,f,s\) are latent variables which denote respectively the pitch index, pitch-shifting parameter and instrument source. In Eq. (1), \({\varvec{P}}\left(t\right)\) is the energy distribution of the spectrogram, which is known from the input signal. \({\varvec{P}}\left(z-f|s,p\right)\) denotes the spectral templates for a given pitch p and instrument source s with f pitch shifting across the log-frequency. \({{\varvec{P}}}_{t}\left(f|p\right)\) is the log-frequency shift for each pitch on the time frame t, \({{\varvec{P}}}_{t}\left(s|p\right)\) represents instrumentation contribution for the pitch in the time frame t, and \({{\varvec{P}}}_{t}\left(p\right)\) is the pitch contribution which can be considered as transcription matrix on the time frame t. Since there are latent variables in this model, the expectation maximization (EM) algorithm [20] is often used to iteratively estimate the corresponding unknown variables.

In the Expectation step, the Bayes’s theorem is adopted to estimate the contribution of the latent variables p, f, s for reconstruction of the model:

In the Maximization step, the posterior of Eq. (2) is used to maximise the log-likelihood function in Eq. (3), which leads to the update of Eqs. (4)–(7). As suggested in [30], this step can converge after 15–20 iterations. The final result of the pianoroll transcription is derived by \({\varvec{P}}\left(p,t\right)={\varvec{P}}\left(t\right){{\varvec{P}}}_{t}\left(p\right)\):

This section is the core of the proposed MPE system where music theories in terms of the pattern of beat length and assumption of equal energy between mixed monophonic and polyphonic music pieces are used to guide the model for the extraction of the multiple fundamental frequencies from a mixture of music sources.

For a given piece of music, the time domain representation is illustrated in the input module in Fig. 1. The results of CQT and SI-PLCA are given in Fig. 2a and b, respectively. Upon observing Fig. 2b, the fundamental pitch and its harmonics have been highlighted by the shaded black and grey strips. However, there is considerable noise and redundant information represented by small and grey dots which may be misconstrued for pitches at lower frequencies. Furthermore, the white gaps in the black and grey strips indicate frequency information that has been lost in the analysis. This suggests that the consistency of fundamental pitch is insufficient if considered frame by frame (each frame was set to 10 ms). To address these inconsistencies, the HSD method is proposed followed by a note tracking process (Fig. 1).

The proposed HSD includes two main stages. In the first stage, the pianoroll transcription \({\varvec{P}}\left(p,t\right)\) is normalised into \(\left[\mathrm{0,1}\right]\) by using the following max-mean sigmoid activation function [32]:where \({\varvec{P}}{\varvec{N}}\) represents the normalised \({\varvec{P}}\left(p,t\right)\). By applying a mean filter in Eqs. (8) and (9), the spectrogram can be smoothed. For extreme values which are too large or too small than expected, they can also be rationalised. For any \({\varvec{P}}{\varvec{N}}\), the value of \({{\varvec{P}}{\varvec{N}}}_{t}\) at time t can be expressed by Eq. (10).

Inspired by the music theory that most high-order harmonic components are in the high-frequency range with low amplitude [17], a two-step hard constrain is used to remove most of the high-frequency components, noise and redundancy. First, a fixed threshold \({TH}_{1}\) is applied in Eq. (11) to remove small values. Based on the characteristic of sigmoid function (Eq. (8)), \({TH}_{1}\) is set to 0.5. Finally, the filtered result \({\varvec{P}}{\varvec{F}}\) of the whole frames is obtained and shown in Fig. 3a.

In the second step, the statistics of the beat length is used to guide the removal of noise and redundant information. According to the cognition of music perception, most notes in musical rhythms have a large number of crotchets and quavers, but fewer numbers of semiquavers and demisemiquavers [33]. The rate of occurrence of different notes in the BACH10 database was observed and measured according to the ground truth. A plot was generated of time vs. rate of occurrence in Fig. 4, with the labelled fractions (i.e. \(\frac{1}{2},\frac{1}{4},\frac{1}{8},\frac{1}{16},\frac{1}{32}\)) denoting minim, crotchet, quaver, semiquaver and demisemiquaver, respectively. Figure 4 illustrates that the rate of occurrence of crotchets and quavers is larger than that of the demisemiquavers, semiquavers and minims. Especially, the number of demisemiquavers and semiquavers is extremely low. Furthermore, if the length of a semibreve is defined as \(\tau\), the length of a demisemiquaver is \(\tau /32\). Any notes shorter than a demisemiquaver will be removed in \({\varvec{P}}{\varvec{F}}\) before any further processing in the second stage.

In Fig. 4, a peak value is identified at the initial time steps of the simulation, and this may be due to two reasons. Firstly, manually played music may contain some timing errors, for example, holding a note for its precise duration for every note in the piece may be impossible. Secondly, ornaments such as vibrato and glissando may be mistakenly performed despite not being present on the music score. The length of such vibrato and glissando is equal to a demisemiquaver or lower [34]. To extract more of the main body of multiple pitches, factors such as human playing habits or ornaments are ignored in the proposed work. Relevant results given in “Experimental results” demonstrate that the multiple pitches are highlighted whilst removing most of the unwanted noise.

After filtering the amplitudes from PLCA, the HSD framework was proposed to detect the fundamental pitch in the second stage. The flowchart in Fig. 5 outlines the process of HSD, and Table 1 lists the description of each parameter. As described in the flowchart in Fig. 5, the output from previous steps will be analysed in two domains, i.e. pitch domain \({\varvec{P}}{\varvec{D}}\) and energy domain \({\varvec{E}}{\varvec{D}}\). In this context, each frame of \({\varvec{P}}{\varvec{F}}\) is split into two vectors, \({\varvec{P}}{\varvec{D}}(n)\) and \({\varvec{E}}{\varvec{D}}(n)\). \({\varvec{P}}{\varvec{D}}(n)\epsilon {\mathbb{R}}^{N*1}\) is non-zero notes index in each frame, \({\varvec{E}}{\varvec{D}}(n)\epsilon {\mathbb{R}}^{N*1}\) is the amplitude of \({\varvec{P}}{\varvec{D}}(n)\), and N is the number of non-zero notes. As seen, the process is only applied once on the non-zero notes rather than the whole frame, because there is no need to analyse those zero-value notes for efficiency.

As seen in Fig. 3b, although most fundamental pitches have been extracted, the notes still show a poor consistency. To improve this, a note tracking method based on the music perception and multi-pitch probability weight was proposed. According to the music theory [33], the occurrence of demisemiquaver is generally quite low in music pieces. As a result, notes with a length shorter than demisemiquaver are filtered out. The averaged rank of the connected pitch group in the rank map is calculated and denoted as \(\overline{R }\). If \(\overline{R }\) is larger than an adaptive threshold \({TH}_{2}\), the pitch group is considered a harmonic and will be skipped from the analysis. As the polyphonic music pitches vary over time, the \({TH}_{2}\) will also change accordingly. To account for this change, a fitting function was generated for \({TH}_{2}\) (Fig. 7a), which is adaptive to the number of notes \(x\in \left[\mathrm{1,12}\right]\) for each frame, as given:

The fitting curve of \({TH}_{2}\) is obtained by minimising the fitting error between ground truth and our estimate. Figure 7b displays the note tracking results where most of the noise and the inconsistencies have been filtered out. The result has also achieved a similar profile to that of the ground truth data.

To validate the effectiveness of the proposed approach, the first dataset used for evaluation is the MIDI Aligned Piano Sounds (MAPS) [37], in which all music pieces are recorded in the MIDI format initially and then converted into “.wav” format. MAPS also have differently purposed subsets such as monophonic excerpts and chords. For this case, only one subset is used which includes polyphonic music pieces. In addition, there are several instruments and recording conditions in MAPS. The “ENSTDkCI” is chosen as the music played using a real piano rather than an acoustic one, i.e. a virtual instrument, and recording occurs in soundproofed conditions. The second dataset is BACH10 [38], which contains 10 pieces using violin, clarinet, saxophone and bassoon from J.S.Bach chorales, where each piece lasts approximately 30 s. The third dataset is TRIOS [39], which is the most complex one among the three as it contains five multitrack chamber music trio pieces. The sampling rate for all music pieces is 44,100 Hz.

For objective assessment, the most commonly used frame-based metric, F-measure (\({F}_{1}\)) [40, 41], is adopted. It combines the positive predictive value (PPV, also namely precision) and the true positive rate (TPR, also namely recall) for a comprehensive evaluation as follows:where \(TPR=\frac{{T}_{p}}{{T}_{p}+{F}_{n}}\), \(PPV=\frac{{T}_{p}}{{T}_{p}+{F}_{p}}\), and \({T}_{p}\), \({F}_{p}\) and \({F}_{n}\) refer respectively to the number of correctly detected \({F}_{0}\), incorrectly detected \({F}_{0}\) and missing detection of the \({F}_{0}\). Specifically, these three components can be calculated by comparing the binary masks of the detected MPE results and the ground truth.

Table 3 shows the quantitative assessment of 11 benchmarking methods on MAPS, BACH10 and TRIOS datasets. We divide all benchmarking methods into two categories: shallow learning method and DL method. Weak learning methods include a traditional machine learning model or a prior knowledge-based model whereas DL methods include deep neural networks and deep convolutional neural networks.

Many MPE approaches select a pair of methods from CQT, PLCA, equivalent rectangular bandwidth (ERB) and NMF for pianoroll transcription. Therefore, two of the most representative methods, i.e. CQT + PLCA proposed by Benotos and Dixon [31] and ERB + NMF proposed by Vincent et al. [42], are chosen for benchmarking. In Table 3, Benetos et al. [43] and Vincent [42] can produce the second best performance on the MAPS and TRIOS datasets, respectively, which validates the effectiveness of CQT + PLCA and ERB + NMF. However, due to the lack of efficient harmonic analysis, the performance of both methods is inferior to the proposed HSD method. Unlike the methods from Benetos and Vincent, other methods adopt different ideas for MPE. SONIC [44] proposed a connectionist approach where an adaptive oscillator network was used to track the partials in the music signal. However, without a matrix factorization process, its performance is limited on the three datasets. Su and Yang [8] proposed a combined frequency and periodicity (CFP) method to detect the pitch in both frequency domain and lag (frequency) domain. The CFP method in Table 3 gives the best performance on the BACH10 dataset, but relatively poorer results on the other two datasets. The main reason here is possibly because the music pieces in the MAPS and TRIOS datasets have more short notes than those in the BACH10 dataset, and CFP has the limited ability for detecting the short notes but exhibit less errors for continuous long notes. Furthermore, the assumption of CFP does not hold for high-pitch notes of piano, as both MAPS and TRIOS have many piano music pieces. In addition, the music pieces in the MAPS database contain multiple notes in most frames, which have led to extra difficulty for polyphonic detection. However, the proposed method can still successfully solve this problem by effectively analysing the relationship of the position and energy between the fundamental frequency and harmonic frequencies for the notes. As a result, the performance of the proposed method on MAPS is the best, which is 8% higher than that of CFP. Klapuri [3] proposed an auditory model–based \({F}_{0}\) estimator, and Duan [38] proposed a maximum-likelihood approach for multiple \({F}_{0}\) estimation, but both methods result in inferior performance compared to the results achieved by Benetos et al. [31, 43], Vincent et al. [42] or CFP [8]. Furthermore, Klapuri’s [3] and Duan et al.’s [38] methods lack an effective pre-processing stage (i.e. TF representation and matrix factorization) or harmonic analysis, which is the main reason why their overall performance is less effective in comparison to ours.

The proposed method was also compared with four deep learning–based supervised approaches on MAPS dataset. Due to lack of publicly available source codes, only the data that was reported in the original paper was duplicated for comparison. The first two methods are proposed by Sigtia et al. [15], which are mainly based on the music language models (MLMs). However, due to insufficiently labelled data in the existing polyphonic music databases for training, such limitations have affected further analysis of DL-based approaches. Furthermore, the MLM model is not robust to ambient noise, whereas music pieces in reality generally contain a lot of ambient noise. This has resulted in DL-based methods failing to fully analyse the inner structure of the music pieces. As a result, DL-based methods cannot achieve the same performance as the HSD method or some of the other unsupervised methods such as Benetos et al. [43] on the MAPS dataset. Su [40] and Kelz [41] also proposed DL-based methods for AMT. Although better than [15], their performance is still not ideal as there is insufficient music knowledge support embedded. To this end, more music theories should be introduced for improved AMT.

In summary, referring to Table 3, the proposed method yields the best results on both the MAPS and TRIOS datasets, also the second-best in BACH10 according to \({F}_{1}\) value, thanks to the guidance of music cognition. However, the method can still be improved, especially for reducing the computation cost. As it takes 2 min to process a 30-s music piece, this is longer than some other methods. In addition, although the profile of the real polyphonic note is close to the expected mixed monophonic note, as shown in Fig. 6e, f, there are still some differences in the final values of the monophonic and polyphonic profiles which can be further improved.

In this section, the contribution of several major stages in the proposed MPE system is discussed, where the performance of each stage is evaluated on the MAPS dataset in terms of the precision, recall and \({F}_{1}\). To calculate these three metrics, the result of each stage is normalised by using Eqs. (8) and (9), and the results are binarized with a fixed threshold value of 0.5.

We generalize our proposed MPE system into four key stages detailed as follows:

Table 4 illustrates the details of the system configurations. By combination of different key stages, the corresponding system is built up for evaluation. Each stage has specific components which are indispensable to the results of the system. Stage A shows the highest recall and lowest precision after applying CQT and SI-PLCA. The presence of \({F}_{0}\) and harmonics is all detected; however, many amplitudes are concentrated in higher frequency (harmonic) regions which inhibits the identification of \({F}_{0}\). After combining stage B, the recall value decreases by 0.03%, but the precision value increases by almost 3%. This is mainly due to the removal of noise in HSD. In stage C, the core of the MPE system contributes to an increase of nearly 30% for precision and 15–18% for \({F}_{1}\) compared to previous combinations. Finally, after applying the proposed note tracking step (stage D), the recall value is further improved by 5.5% which leads to the final \({F}_{1}\) value improved by 3.8% compared to the previous stage.

In our proposed MPE system, CQT is employed to model the human cochlea perception. However, cochlea perception is not always constant in Q. Therefore, apart from CQT, the equivalent rectangular bandwidth (ERB) method is also widely used for time–frequency transform [42]. As most ERB methods are actually based on the Gamma tone filter-bank to model the human auditory system [45], it decomposes a signal and passes it through a bank of gamma tone filters, equally spaced on the equivalent rectangular bandwidth (ERB) scale. However, ERB methods may not be necessary to produce better MPE performance than CQT. To further validate this assumption, we have combined CQT [27] and ERB [42] pair-wisely with PLCA [43] and NMF [42] to form four hybrid methods, i.e. CQT + PLCA, CQT + NMF, ERB + PLCA and ERB + NMF, for quantitative analysis in terms of the precision-recall, ROC, F-measure curve (Fig. 8), AUC, MAE and maxF (Table 5). Here AUC, MAE, and maxF denote respectively the area under the ROC curve, the mean average error and the max value of F-measure curve. These three criteria have the same importance. As seen in Fig. 8, the ERB + NMF and CQT + PLCA show comparable results; both outperform the other two methods. In Table 5, although ERB + NMF gives the best maxF value, CQT + PLCA gives the best AUC and lowest MAE, indicating a smaller false alarm. Therefore, CQT + PLCA is the best among these four methods, which is also the main reason why it is used in our proposed MPE system.