Automated composer recognition for multi-voice piano compositions using rhythmic features, n-grams and modified cortical algorithms

Starting with a single track raw midi file, the feature extraction stage aims to construct a feature vector representing the midi file.

Naturally, the first step is to compile the data from which the features are generated. The ultimate goal is to create a representation of audio files for the performed music when recorded using microphones. This would require performing automatic transcription, i.e., identifying the sequence of notes and their characteristics such as the dynamics, rhythms, pitches, etc. However, the ability to perform automatic transcription is still an immature field due to the fact that advanced auditory signal processing techniques are the focus of researchers worldwide especially for polyphonic music [1, 4, 23, 30, 31]. In addition to that, as we mentioned earlier, a recording of the same musical composition might be very different if interpreted by different performers. Hence to classify a recording based solely on the composer style, one has to develop a scheme to “deconvolve” the interpretation of the performer which becomes a completely different problem. For these reasons, the data set is extracted from MIDI files (representations of the sequential progression of musical notes and their characteristics) using an available Matlab toolbox [8].

The most important parameters representing a musical piece are the pitch and duration of each note in a midi sequence. However, since the same musical piece might be played in a different tonality or even tempo; therefore, a relative representation of the pitch and duration is preferable compared to the absolute representation that was proposed in [23]. For the ith and \((i-1)\)th notes having a pitch \(p_{i}\) and \(p_{i-1}\) and a duration \(t_i\) and \(t_{i-1}\), respectively, we construct a representation \(P_i\) for the pitch difference and \(T_i\) for the time duration difference as shown in Eqs. 1 and 2.

After data extraction, features are generated. These features should characterize the data in a compact and efficient form for our purpose. As suggested in [32], the highest pitched notes (skyline) reveal the melody of a certain piece, while the progression of the base notes (groundline) gives a good indication of the style of the composer. Therefore, we start first by extracting the melody and the bass notes and then the corresponding pitch and duration n-grams as proposed in [23]. However, in addition, we propose another set of features which characterizes the rhythmic style of the composer. These features quantize the number of notes that have onsets on a complete beat, half of a beat, quarter of a beat, one third of a beat, etc. This captures the style of the composer in placing notes on beats or between beats (the so called syncope) or even triplets. We included in section “Illustrative examples” an illustrative example of our proposed feature generation.

Figure 1 shows a detailed setup for the feature generation stage from a midi file. First, the note matrix (see Table 1) is constructed, and then the highest and lowest note sequences are extracted. The note sequences are then filtered based on a computed threshold defined as the average of the centroids of the highest and lowest note sequences. The process is repeated resulting in a representation of the piece consisting of a robust melody and bass extraction. The repetition of the process is particularly useful to categorize a solo note sequence as melody or bass. The final feature vector for the input midi file is a vector containing occurrences of N possible n-grams in addition to the onset beat features we propose. It is important to note that N varies with each midi file. Hence to obtain a fixed length feature vector, automatic zero padding is employed where non-existing n-grams are added with 0 occurrences.

To show the approach of our melody/bass extraction algorithm, we use (as an illustrative example) the first seventy beats of Bach’s Prelude No. 17 in A flat major. Figure 2 shows the piano rolls of Bach’s Prelude before and after melody/bass separation.

Next, we illustrate with a simple example the construction of n-grams. We use an evaluation copy of Mozart Software [34] to write the main melody of the Lebanese National Anthem and convert it to a MIDI file. The main melody of the first phrase is shown in Fig. 3 and the corresponding data extracted also known as the note matrix are shown in Table 1. Onsets and Durations (whether in beats or in seconds) give rhythm characteristics, pitches represent the name of the note (60 represents the central C note) and velocity represents the dynamics of the note. The data are generated as follows: unigrams are constructed using Eqs. 1 and 2. Then, pitch and duration n-grams extracted are constructed. For example, 3-grams are formed for the sample mentioned. Pitch (duration) unigrams and 3-grams of the sample are shown in Tables 2 and 3, respectively. Finally, the rhythm feature vector for the sample is constructed and shown in Table 4.

CA are a biologically inspired machine learning (ML) approach which structure (columnar organization) has been proposed by Mountcastle and Edelman [7]. The model mimics aspects of the human visual cortex that recalls sequences stored hierarchically in an invariant. A mathematical formulation and a computational implementation of this model were further developed by Hashmi and Lipasti [13‐16]. The model consists of an association of six-layered structures of different thickness constituted of a very large number of columns strongly connected via feed-forward and feedback connections. The column, considered as the basic unit in a cortical network, is constituted by a group of neurons that share the same input. An association of columns is referred to as hyper-column or layer. The connections in the network occur in two directions: vertical connections between columns of consecutive layers, and horizontal connections between columns within the same layer. Though CA can be considered as a type of deep neural networks (DNNs) due to its hexa-layered architecture, two main differences with DNNs can be noted at the levels of structure and training: traditional DNNs employ a feedforward architecture while feedback connections are adopted in CA, and while DNNs rely on a backpropagation algorithm for their training, CA aims at forming unique representations in its supervised learning scheme [12].

A mathematical representation for CA, as we developed based on the description provided in [7], is detailed below.

The nomenclature we adopt in our work is as follows: \(W_{i,j,k}^{r,t}\) represents the weight of the connection between the jth neuron of the ith column of layer r and the kth column of the previous layer \((r-1)\) during the training epoch t. For the rest of this paper, scalar entities are represented in italic, vector entities in bold, while underlined variables represent matrices [11]. The weight matrix representing the state of a column composed of M nodes at the training epoch t is \(W_{i,j}^{r,t}\) represents the weight vector of the connections entering neuron j, of column i in layer r which can be written as: \(L_{r-1}\) is the number of columns in the layer \((r-1)\) and the subscript T stands for the transpose operator. Expanding \(W_i^{r,t}\) yields:Defining \(Z^{r,t}\) as the output vector of layer r for epoch t and \(Z_i^{r,t}\) the output of column i, we can writeThe output of a neuron is the result of the nonlinear activation function f(.) in response to the weighted sum of the connections entering the neuron. The output of a mini-column is defined as the sum of the outputs of the neurons in this column: \(Z_{i,j}^r\) is the output of the jth neuron constituting the ith column of the rth layer and the activation function is defined as follows:Note that \(Z^0\) refers to the input vector \(X=[X_1, \dots , X_D ]\) and we denote by D the size of the input vector which corresponds to the number of features or the data dimensionality. \(\tau \) is a tolerance parameter empirically selected and it is assumed constant for all epochs and columns. This parameter sets the tolerance of the activation function to large input values, the larger this value is the less sensitive the activation function becomes.

The training of a cortical network has three phases:

One of the advantages of CA is its ability to extract and learn the aspects of the data in a hierarchical manner. This makes CA a possible candidate for feature extraction or reduction, which to the best of our knowledge has not been investigated yet [22]. As described previously, one goal of CA training is to train cortical columns to fire for particular aspects of the patterns during the feedforward unsupervised learning. In addition, during the feedback supervised learning, invariant representations are created on all levels of the network in a hierarchical manner. As a result, a repetitive firing scheme is observed for a particular class of the data. The weights of the input layer directly connected to the feature vector are, therefore, an indication of the significance of the corresponding features. Our method employs hereby denoted by CA-FR, the input layer weights to rank the feature vector in terms of significance, discarding the lowest ranked features, allowing for a higher discriminative performance of the algorithm and a robust model of the raw data. The following algorithm illustrates our approach hereby denoted by CA-FR:

In this section, we summarize the experimental results obtained. We perform our experiments on a database of note files we compiled from the Humdrum project library [19] described in Table 5. The era of each composer as well as the total number of beats for each composer are also shown in this table.

Our experiments are designed to test the features we proposed, our feature reduction technique, as well as CA as a classification technique for the recognition of artists. Input weights of the network were used to discern the least significant features (corresponding to the weakest input weights). A fraction of these features is eliminated at each iteration. Thus, we perform four experiments using our dataset:For each of the four experiments, we subsequently eliminate a fraction of the least significant features. The resulting subsets are fed to three recognizers: CA, SVM one vs. all using Gaussian Radial Basis Function (RBF) kernel and a one nearest neighbor classifier (1NN). SVM and 1NN were chosen for comparison purposes since they are widely adopted in a multitude of machine learning applications [27]. We evaluate the performance of these classifiers in terms of recognition rate. We used a cortical network constituted of 6 layers starting with 2000 columns in the first layer and decreasing this number by half between consecutive layers, each column consisting of 20 nodes. This structure was experimentally proven to perform well on a variety of databases [11]. Simulations were executed using Matlab R2011a, on an Intel core i7 CPU at 2 GHz under Windows 7 operating systems. The CA network is deployed using the CNS library, a framework for simulating cortically organized networks, developed by MIT [28].

Table 7 shows the recognition rates obtained using the three classifiers described above for different percentages of the initial dataset reduced using PCA and CA-FR as described in the experimental setup, while Table 8 shows the required training time. The best recognition rate for each experiment is shown in bold. All results are based on a fourfold cross validation scheme. Table 9 shows the recognition rate obtained using experiment 1 settings on 0.1% of the features for a 2 two-composer classification task. Composers are arranged according to their year of birth to reflect their closeness in eras. In addition, Table 10 provides the mean, standard deviation, maximum and minimum recognition rate obtained for every experiment (for all reduction rates) as well as the overall statistic for every classifier. Table 6 compiles results from recent work on these composers.

The following can be concluded from our experimental results. Using the entire dataset with the proposed added features, the best recognition rate achieved is 88.4% using CA as classifier compared to 70.3 and 60.6% using SVM and NN, respectively, and thus an improvement of CA of 18.1 and 27.8% over SVM and NN, respectively.

Using only 0.1% of the features including the proposed attributes, the best performance achieved is 94.9% using CA as a classifier, 72.8% using SVM and 62.2% using NN. This is a strong indicator of the superior performance of CA’s classification compared to classical pattern recognition classifiers.

For the same reduction rate of 99.9%, our proposed added features introduced an improvement from 86.4 to 90.4% using PCA and from 91.1 to 94.9% using CA-FR, showing the significance of our proposed added features in discriminating composers. In other terms, an improvement of 4% is obtained using our proposed features and PCA for feature reduction (Exp. 3 vs. Exp. 4) and 3.8% using our proposed features and CA-FR for feature reduction (Exp.1 vs. Exp. 2)

For the same percentage of features (0.1%), the best recognition rate obtained is 94.9% using CA as classifier and our feature reduction algorithm while PCA led to an accuracy of 90.4% for the same reduction rate. This shows CA-FR’s ability in extracting the most significant features while discarding the less important ones.

Using only the n-grams features and our feature reduction technique resulted in a slightly better performance compared with using PCA on the entire set, indicating the strong capacity of CA-FR of extracting meaningful information and creating discriminative representations of the input data.

The effect of the feature reduction stage is most shown in the classification results: an improvement in accuracy is overall noticed in experiments employing our feature reduction algorithm, a smaller effect of feature reduction can be seen for PCA which shows the robustness of our algorithm in discerning significant features.

While a nearest neighbor classifier requires practically zero training time, the high computational demand of computing distances (specially for high dimensional data) renders its use in an online scheme quite challenging. Conversely while CA and SVM require lengthy training, the testing complexity of both these algorithms is relatively low (SVM only requires the computation of the equation of a line while CA requires only the propagation of the input through the network) and hence are more suitable for an online framework while the training can be performed offline.

Comparing our proposed method with the state of the art results shown in Table 6, a notable improvement in recognition rate can be observed in all one-to-one tasks. This is further intensified in tasks involving composers within the same era such as Haydn and Mozart where the recognition rate saw an increase from 74.7 to 82.9% as well as Beethoven and Chopin where our algorithm achieved 86.1% compared to 63.8% reported in the state of the art.

Comparing classifiers, one can see the higher ability of CA to robustly classify composers by analyzing Table 10: while CA’s higher standard deviation may be regarded as a drawback, its lowest recognition rates rival the best performances of SVM and 1NN consistently for all experiments, hence justifying its use as a classifier.

Comparing the recognition rates observed for one-to-one composer classification tasks, one can see the robustness of our proposed method: the lowest reported accuracy is at 86.1% for a Chopin vs. Beethoven task compared with a 63.8% reported in [20], while the best performance marked a 99.5% accuracy classifying Vivaldi vs. Joplin. Overall, our proposed feature set and feature reduction system showed to be powerful for the recognition of music composers with a 94.4% recognition rate obtained using 0.1% of the entire features (n-grams and proposed attributes), CA for feature reduction and classification.

A deeper look on our proposed framework can be taken by investigating the retained features for the best performance obtained, i.e., for Exp. 1 at 99.9% reduction. As indicated by the input weights of the network, the five most significant features are the ones corresponding to the occurrences of notes with onset on quarter beats, occurrences of notes with onset on complete beats, occurrence of notes with onset on half beats, and 2 n-gram combinations corresponding to bass notes; all of which are proposed by this work. From a music perspective, these results propose that piano composers mostly distinguish their style via the way they deal with bass notes on one hand, and beats/onsets on the other hand. Obviously, melodies play an important role in revealing the style of a particular composer. However, according to our work, the spreading of notes over beats/onsets, together with the development of bass patterns that accompany the melody, has a lot more to say about the style of a particular composer. It is well known that humans, in general, are biased to perceive high pitched melodies much more than the accompanied bass notes. However, given any melody, different composers develop different beats and different bass notes which harmonize with the same melody. This is particularly what distinguishes the style of a composer as the retained most significant features propose. Furthermore, looking at the results of one-to-one classification (Table 9), confusion rates ranged from 17.1% (Mozart vs. Haydn) to 0.5% (Vivaldi vs. Joplin) with Joplin being the least confused composer, due to his belonging to a different era than all other composers. The least accurate prediction rates were found for the tasks of distinguishing Haydn vs. Mozart, and Beethoven vs. Chopin, consistent with the similarities in style between the mentioned composers. In fact, it is known in the music society that Haydn and Mozart were friends with the latter being inspired by the former [5]. Additionally, while both Beethoven and Chopin have their own styles, some pieces of these composers possess some similar features (particularly, Beethoven’s Adagio of op. 106 or the Arioso in op. 110 has a chromatic style, a feature Chopin has been known for).