Data properties and the performance of sentiment classification for electronic commerce applications

To implement machine learning based sentiment classification, the following standard of bag-of-words framework is usually used. Let {f ₁, f _{2 ,  … .} f _m } be a predefined set of m words (features) that can appear in a document and n _i (d) be the number of occurrences of f _i in document d. Then, each document d is represented by m-dimensional document vector \( \overrightarrow{d}=\left({n}_1(d),{n}_2(d),\dots, {n}_m(d)\right) \). For example, if the predefined set is defined as {this, that, is, a, car, bicycle}, then the document “this is a car” can be presented as (1,0,1,1,1,0).

This bag-of-words framework is used for two step of MLA; training and prediction. As depicted in Fig. 1, labelled documents in training dataset are converted to document vectors during training. Basically, all documents in training dataset have one of two label; positive or negative. Document vectors and their label should be used to classify un-labelled documents in test dataset. During prediction, a document in test dataset is converted to a document vector. This document vector is then fed into the model, which generates predicted labels whether it is positive or negative (Bird et al. 2009).

In the previous research, Naïve Bayesian (Kang et al. 2012; Yoshida et al. 2014), support vector machine (Mullen and Collier 2004) and decision tree (Sui et al. 2003) have been used for implementation of sentiment classification and these algorithms are reported to be effective for sentiment classification in the literature (Forman 2003; Dhillon et al. 2003; Sebastiani 2005; Wan et al. 2012; Ur-Rahman and Harding 2012). In this study, we also use these three algorithm for the experiment of impact of data properties on sentiment classification performance.

Naïve Bayes is one of the most popular MLA as it is easy to implement without any complicated iterative parameter estimation schemes (Wu et al. 2008). Based on the bag-of-words model, Naïve Bayesian based sentiment classification defines the likelihood of a document (d) to be positive or negative as a sum of total probability over all mixture components, i.e., P(d) = ∑ _j P(d| positive)P(positive) for positive; where P(positive) is the probability of the positive and P(d| positive) is the probability of the document belonging to positive. For balanced training dataset, P(positive) and P(negative) are equal to 0.5 as the equal number of documents are used for positive and negative. To compute the likelihood of being positive or negative for given document, Naïve Bayesian approach applies the so-called “naïve assumption” that all words are independently used in all document, (Melville et al. 2009), which implies that P(w _i ) = P(w _i | w _j ) where w _j can be any other words. Based on this assumption, the probability of a document d being generated in positive is P(d| positive) = ∏ _i P(w _i | positive), where i is the number of words in a document.

The Naïve Bayes classification rule uses Bayes’ theorem to compute the probabilities of a document belonging to class c _j as follow,and the label with the highest likelihood is predicted, i.e.,where label = {positive, negative}.

In this research, we adopt Multinomial Naïve Bayes as a classifier as it works well for text data that can easily be turned into numerical data, such as word-counts in a text.

Support vector machines (SVM) is considered as one of the best text classification methods (Xia et al. 2011). SVM is a statistical classification method based on the structural risk minimization principle of the computational learning theory. In sentiment classification case, SVM seeks a classification surface that can separate positive and negative document vectors from training dataset. All documents in training dataset can be vectorised using bag-of-word framework and then these document vectors can mapped into vector space as depicted in the Fig. 2.

The process for finding a hyperplane, presented by vector \( \overrightarrow{w} \) in Fig. 2, corresponds to a constrained optimisation problem to maximise the margin and the solution can be written as:where α _j ’s are obtained by solving a dual optimisation problem and c _j  ∈ (1, −1), which means positive and negative. Those \( \overrightarrow{d_j} \) such that α _j is greater than zero are called support vectors, since they are the only document vectors considered to find the hyperplane\( \overrightarrow{w\ } \). Classification of test document vector consists simply of determining which side of \( \overrightarrow{w\ } \)‘s hyperplane they fall on (Pang et al. 2002). In this study, we use radial basis function (rbf) as a kernel function for classifier.

Decision Trees Classifier (DTC) is a non-parametric supervised learning method used for classification and non-parametric regression analysis. The goal of DTC is to create a model that predicts the value of a target variable by learning simple decision rules inferred from data features. DTC has been used for sentiment analysis as it is easy to understand the algorithms and its result (Pang et al. 2002). However, it is not commonly applied to long document classification because a long document tends to be converted into a high dimensional document vector.

DTC for sentiment classification usually work top-down, by choosing a feature (i.e., word) at each step that best splits the documents in training dataset (Rokach and Maimon 2005). For metrics to choose features for each step, we use Gini impurity (Kuh and De Mori 1995), which is a measure of how often a randomly chosen document from training dataset would be incorrectly labelled if it were randomly labelled according to the distribution of labels in the subset (see Fig. 1 for labelling). Gini impurity can be computed by summing the probability f _i of each item being chosen times the probability 1 − f _i of a mistake in categorizing that item. It reaches its minimum (zero) when all cases in the node fall into a single target category. To compute Gini impurity for a training dataset, let f _i be the fraction of items labelled with value i (in this case, positive and negative) in the set.

The SOA is based on identifying and selecting sentiment words in test documents (Wang et al. 2014). The main idea of this approach is to classify the sentiment of words in a document to infer its semantic orientation, i.e. whether the document has positive or negative opinion. To classify the sentiment of words, this approach usually uses external data sources such as corpus that has massive text data containing sentiment expression and dictionary that shows the polarity of large number of words. The corpus and dictionary can provide either the polarity score between −1 and 1 (−1 means extreme negative and +1 means extreme positive) or sentiment polarity of words (i.e., positive or negative). Studies based on SOA may use learning algorithms to construct dictionary and semantic network between words from corpus therefore can be considered as a learning approach as well. More common approach is to use predefined sentiment lexicons such as WordNet and SentiWordNet that provides lists of sentiment words. Synonyms, antonyms and hierarchies in WordNet with sentiment can be used to determine the polarity of documents (Andreevskaia and Bergler 2006; Das and Bandyopadhyay 2011). SentiWordNet, which was built upon WordNet, also have been used for external resource to score the polarity and classify its polarity (Devitt and Ahmad 2007; Denecke 2008).

Compared with the Naïve Bayes based sentiment classification, a semantic orientation specification based on lexicon of sentiment words is quite simple and intuitive. In this study, we use SentiWordnet (Baccianella et al. 2010; Esuli and Sebastiani 2006) as a lexicon resource for sentiment classification. It is the most popular open sentiment lexicon resource that is used for automatic sentiment classification. Many sentiment classification tasks extract sentimental words directly from SentiWordNet to avoid a manual sentiment lexicon or building new lexicon from the massive data using additional learning approach (Hung and Lin 2013). To classify the sentiment of a given document using the lexicon resource such as SentiWordNet, the elements of a vectorised document should be tagged grammatically, i.e., part-of-speech tag. In corpus linguistic and computational linguistics, part-of-speech tagging (POS tagging) is a process that makes up a word in a text (corpus) as corresponding to a particular part of speech, such as noun, verb, adjective, etc. The automatic POS tagging has a long history in computational linguistic studies and now its tagging accuracy reached to over 97%² (Manning 2011; Toutanova et al. 2003). POS tagged document vectors can be easily scored by comparing adjectives/adverbs/verbs in documents and those in SentiWordNet that contains the polarity scores of words (for example, adjective ‘bad’ has −0.625 and ‘worst’ has −0.75 in SentiWordNet). The pseudo-code representing overall procedure for SOA is presented in Fig. 3.

Though machine learning based approach usually outperforms other approaches, SOA using predefine lexicon is used for several reasons. Firstly, it is one of the most realistic ways to realize sentiment classifier without training dataset. Usually it is hard to have reliable training dataset for classification in practical situation for both researchers and practitioners. Human-rated and tagged data can be seen as the most reliable data but it requires too much effort and human resources. In addition, to replicate the complex classification algorithms is also very challenging while calculating the sentiment score of document based on lexicon is relatively easy to realize. Secondly, as a semantic lexicon such as SentiWordNet contains the general sense of word, this approach is free from domain dependency so it can show the moderate performance regardless of the application domain of test data (Denecke 2009).

As we reviewed in previous two subsections, MLA and SOA has different computational procedure to decide the sentiment of document and each method has their pros and cons accordingly. SOA doesn’t need a training data for its classification but it needs external source instead. In the case of lacking enough training data for given domain, SOA can be applicable with external sources and implemented very easily by matching the words and external sources. Also, SOA is free from domain dependency as it considers the general words for expressing sentiment (Denecke 2009). But SOA can be faced with the problem resolving the semantic of word with multiple meaning (i.e., “mean” as a verb is neutral while its adjective is negative). Another concern is uncertainly regarding the classification of long document with both positive and negative sentiment.

Generally, if enough training data is given, MLA is preferable and known to show superior accuracy than SOA (Aue and Gamon 2005). But it is also known that MLA has domain dependency problem for training classifier. If the training data has different domain from test data, the accuracy can be dropped comparing to the classification with training using training data from same domain (Denecke 2009). Another variable of MLA can be a choice of classification algorithms as each algorithm has its strengths and weaknesses. MLA based on decision tree is very fast and robust to noise but has weaknesses to process long document that can cause complex tree. SVM based MLA can be also robust to noise and good for long document classification due to its strength in high dimensional processing while it takes longer time than other algorithms such as decision tree and Naïve Bayes classifier (Pang et al. 2002). Table 1 shows the summary of pros and cons of each approaches.

Since sentiment analysis has been paid attention from academics and practitioners, many open source library has been developed for various programming language such as Java, python, Ruby, etc. Each library has different algorithm coverages so user can select the library according to the programming language and algorithms. Tables 2 and 3 shows the comparison and algorithm coverages of existing open source library. Note that we only consider the libraries that are still managed and updated by creators as some of libraries stop their update.

Table 4 summarises how this study extends existing studies on the sensitivity of sentiment analysis performance. Compared to existing studies, this research provides comprehensive investigation between data properties and the performance of sentiment classification algorithms covering four algorithms and four different datasets. This study conducts not only the performance comparison among basic sentiment classification algorithms, but also multi-dimensional comparisons based on data properties. Followings are unique findings from this study compared to existing studies. For MLA, we unveiled the existence of optimal word-count of documents for training the classification model and the effectiveness of documents with higher subjectivity as training datasets. The minimum training size for good performance was also validated using four datasets. We also found that the performance of SOA depends on the subjectivity and length of documents in test dataset. The result tells that SOA works well for shorter documents and higher subjectivity. All these findings regarding data properties were absent from previous studies as they focus on simple performance comparison among existing algorithms using limited datasets without data-driven perspective.

Barbosa and Feng (2010) and Pak and Paroubek (2010) shows the impact of subjectivity words on the performance of sentiment classification but only short documents such as Tweets were used in their experiments. Due to the experiment using limited dataset, they cannot investigate potential influence of document length on classification accuracy. Aue and Gamon (2005) have tackled the domain dependency problem of sentiment classification performance and the impact of training size on accuracy but they do not explain a reason why different test datasets produces different classification performances. They do not explain what different data properties their datasets have and how the properties make differences on performance. Ranade et al. (2013) report decreased classification performance when a sentiment statistical scoring model based on sentence length and sentiment words is used for longer documents. However, the results are based on only online debate articles and only SOA is tested.

Similar to this study, there are other studies that also compare existing approaches. Pang et al. (2002) are one of the first scholars who provide the comparison of the performances of different MLA algorithms (Naive Bayesian, Maximum Entropy and Support Vector Machine) by collecting the results from existing studies that use the same dataset (movie reviews). They, however, do not provide the details of their experiment settings such as training size and the properties of the data they use therefore difficult to generalise the findings. They also do not investigate the impact of narrative and objective words on the performance which are one of important contributions of this study.

Tang et al. (2009) provide a referential review on sentiment classification studies as they discuss and introduce related issues and main approaches. They provide a performance comparison based on the reviews of existing studies, however, without discussing factors affecting the performance differences. Though their performance comparison table shows that the same MLA can show different performances with the same IMDB dataset (for example, they showed Naive Bayesian’s performance varies between 65.9 ~ 81.5% according to other studies), they do not pinpoint the factors that cause such performance differences. Vinodhini and Chandrasekaran (2012) report the similar findings with Tang et al. (2009).

Moraes et al. (2013) provides results of empirical comparison between SVM and ANN using same datasets. Their findings show that the performance of SVM and ANN can be affected by word-count of documents in test datasets, however, they do not consider the properties of training datasets such as training size, subjectivity and word-count of documents tht are considered in this study. As the performance of MLA strongly depends on the way of training a classification model, data properties of documents in a training dataset also need to be considered.

As reviewed above, most of existing sentiment classification studies provide simple performance comparison in terms of the “accuracy” of algorithms without considering the role of “data properties and setting” that may cause differences in the performance. Without the details of the data setting and the experiment control, the accuracy of algorithms cannot be replicated as the performances may vary according to the nature of the data used and the way experiments are conducted.

This study investigates the impact of linguistic properties of data such as word-count, size of training dataset and subjectivity of document on the classification performance of two approaches: SOA and MLA. The importance of these properties in sentiment classification is stated in related studies in Table 4, however, none of them provides a comprehensive comparison of performances of different approaches on datasets with different properties.

Based on literature review, the most common data used for sentiment classification in electronic commerce and social media includes news documents, SNS messages, Blog documents, and customer review on products and services. It can be classified according to two dimensions: length and subjectivity. Studies on strategic decision for marketing usually apply sentiment classification to short SNS messages (Hennig-Thurau et al. 2015), product reviews in e-commerce platform (Yang and Chao 2015; Hu et al. 2014), or the Internet forums. News documents and financial columns, which are longer than social media messages and product reviews, can be used for financial decision support and risk assessment (Wu et al. 2014).

The quality and properties of training dataset are of considerable importance for the performance of MLA. Training datasets need to be defined by analysts in such a way that they are typical and representative of each individual class and both quality and size of training dataset are of key importance (Kavzoglu 2009). The size of training dataset is one of the critical properties determining the accuracy of supervised learning. Usually, more training dataset can improve the classification accuracy as too small training dataset can cause over-fitting. However, in practice it is not desirable to organise a very large dataset. To secure a reliable training dataset in a certain domain is a challenging task. For this reason, deciding the optimum size that can produce reasonably high level of accuracy is a common problem in machine learning studies.

A training dataset also needs to have informative and relevant features of their class where they belong to for accurate classification (Kotsiantis 2007). Too much noise and missing features in a training dataset can cause significant diminution in the performance of a MLA based classification algorithm. For this reason, training a classifier with an appropriate dataset is considered as important as implementing sophisticated algorithms in machine learning field and many studies try to improve the performance of classifiers by improving just the quality of used training dataset (Weiss and Provost 2003; Batista et al. 2004; Sheng et al. 2008). Wrongly labelled training dataset (e.g., some positive documents in negative training dataset, and vice versa) causes a poor classification performance. The use of a training dataset with human-generated labels can solve this problem but not realistic when a massive training dataset is needed (Gamon 2004). One of the alternative approaches is to use polarized reviews from e-commerce platforms; for example, 1 star reviews for negative and 5 star reviews for positive training datasets using a 5 scale rating system (Pang and Lee 2005).

The length of documents in training dataset may be significant in determining the number of informative features. Long documents are likely to have many words that have sentiment even though we cannot confirm that most of sentiment words are informative for classifying the sentiments of the whole documents. The uncertainty of consistency between the sentiment of single words and that of whole document can be found in some theories related to the human utterance and behaviour. By the Politeness Principle (Leech 1983 ), people tend to mix opposite opinion to show their politeness and to emphasize their opinion. The example:

“This movie has a fantastic scale and a perfect location for a fantasy movie. But there’s no theme so I can’t understand what the director want to say. The plot is also awful. I do not want to recommend this movie to my friends.”

This review is obviously negative review but the paragraph has both positive and negative sentiment. By POS tagging, the words that have semantic orientation will be added as features for classifier but all the words are not informative. This example shows that why we need to analyse the effectiveness of long document as a training dataset.

In SOA, using a predefined semantic lexicon, the property of the test data can affect the accuracy of classification. The document with simple expression of sentiment is more likely to be classified correctly as people tend to use short words and use symbols in their comments to express their opinion (Khan 2011). If a document has both positive and negative words, the polarity score of the document may not be good enough for classification as the score is usually derived from the summation of scores of each individual word (Kim and Song 2013).

One of the difficulties associated with sentiment classification of web contents is that datasets tend to be highly imbalanced as there is general tendency that users are willing to submit positive reviews while hesitant to submit negative ones (Liu and Yu 2014). For this reason, we select balanced datasets such as IMDB movie reviews (Maas et al. 2011) that are used for many sentiment classification studies due to its balanced amount of data between positive and negative reviews. Twitter datasets (Mukherjee and Bhattacharyya 2012) and hotel review datasets (Lu et al. 2011; Pontiki et al. 2014) has been used in the previous studies and verified to be balanced datasets. Amazon review datasets contain reviews for small electronics and collected using script crawling for this research. All the repeated data has been filtered to avoid redundancy and non-English data also has been removed using NLTK (Bird 2006) language detection function. For the data from IMDB, Hotel Review, and Amazon, we assigned the positive and negative to the document following to the star rating form review authors as Pang et al. (2002) and Pang and Lee (2005) did. Twitter dataset was adopted from (Mukherjee and Bhattacharyya 2012)) as it contains manually tagged sentiment but we eliminated hashtag to make same experiment environment as the other review dataset had no hashtag.

The datasets cover various domains and contexts of e-commerce and social media as shown in Table 5. For the classification models, we adopt a SentiWordNet based word scoring for SOA and Naïve Bayesian, Decision Tree, and Support Vector Machine classification model for MLA. For the implementation of all classification methods and corresponding experiments, Python 3.0 (v.3.4.3) was used with text processing and machine learning libraries such as scikit-learn (v.0.71.1)³ (Pedregosa et al. 2011), Anaconda (v.2.3.0 for 64 bit),⁴ and CliPS (v.2.6) (Smedt and Daelemans 2012), etc.

To investigate the sensitivity of the performance on word-count in test documents, we split the documents in the datasets into several groups having different word-counts except twitter dataset as it contains only short documents due to the word-count limitation of the twitter service. Table 6 shows how the accuracy of SOA changes as the word-count increases. The overall accuracy of the SOA was around 0.65 ~ 0.75, which shows the similar level of performance with other studies that use the same approach (Ohana and Tierney 2009).

From the experiment results, we can see that SOA based classification shows better accuracies when the test documents have less words. The accuracy of the results of documents with less than 200 words is significantly higher than those in other groups except hotel dataset. Hotel datasets results show the higher accuracy for longer document but it has small portion for longer document. For IMDB and amazon datasets, the accuracy of short document (word count 0 ~ 100, 101 ~ 200) show higher accuracy than other longer documents. So we can conclude that SOA show better performance for those with less than 200 words than other documents.

Though the results do not imply the inverse linear relationships between word-count and accuracy, we can conclude that SOA is more appropriate for shorter documents (less than 200 words count).

Next experiment was designed to test the impact of subjectivity of test documents on the performance of SOA. In this experiment, we tagged POS and found the words that had semantic orientation based on SentiWordNet and finally found the document’s subjectivity by averaging the subjectivity scores of the tagged words. Documents that have many strong polarity words such as “best” or “extremely” were expected to have higher subjectivity score.

As shown in the experiment result in Table 7, SOA shows the better performances for documents with higher subjectivities. That is, if the authors of documents use a strong negative words to express their sentiment, only few positive words might be found from the documents. This makes the document with strong subjective words easier to be correctly classified.

The last experiment is to investigate the performance difference between MLA and SOA with regards to the properties of test data. Without controlling the data properties of training dataset, 5% of test data size are used for extracting features for each positive and negative (IMDB – 500 training and 10,000 test documents, Hotel review – 400 training and 8000 test documents, twitter – 200 training and 4000 test documents, and Amazon review – 300 training and 6000 test documents for each positive and negative). All experiments are performed 20 times with random sub-sampling, which is equivalent to 20-fold validation, to derive reliable results. The distribution of test documents with regard to the subjectivity and word-count is summarized in Tables 9 and 10 shows the accuracy comparison based on the test data properties.

Multinomial Naïve Bayes and Support Vector Machine show better performances than SOA for all dataset cases. Both approaches show better accuracy for high subjectivity test data than for low subjectivity test data. MLA shows moderate performance for the test documents with more than 100 words. It fails to show good performance for short documents with less than 100 words while the performance of SOA is best among other for that range. Short documents containing small but consistent sentiment words are more likely to be correctly classified using SOA as it can cover wide range of words based on dictionary while MLA misses some words due to the limitation of training dataset size. For a training dataset with more than 200 words, MLA shows the better performance than SOA. In this case, MLA is more effective because the longer documents are likely to have both positive and negative sentiment words that causes misclassification. This indicates that only few features play an important role for correct classification for long documents.

The running time for each experiments is presented in Table 11. Multinomial Naïve Bayes classifiers take less running time for training and test in all cases while SVM take longer time for both training and test due to the high dimension problem.

Key findings from experiments are summarised in the Table 12.