Reliability and Perceived Value of Sentiment Analysis for Twitter Data

Social media offers rich data sources for companies that want to understand how they are perceived by their stakeholders. Sentiment analysis over Twitter can produce information about people’s feelings toward their brand, business, and directors (Saif et al. 2012). Based on this information, companies can take actions to enhance their customer experiences and perceived brand value. This study investigates the reliability and perceived value of two sentiment analysis tools developed to understand Finnish language, in contrast to human evaluators. For this purpose, a dataset of tweets from a Finnish software company was collected. For evaluating reliability Krippendorff’s α (Krippendorff 2007) is computed. Perceived value of the automatic and human evaluator classified sentiment is evaluated by interviewing the case company representatives. The results point out that the analysis carried out by the human evaluators was perceived more valuable by the company representatives than the automatic analysis, due to different granulation level of the analysis. Compared to the automatic analysis, the human evaluators were able to put the identified emotions from the tweets better into a context, which in turn diminished the potential misinterpretation of who was the target of the most negative tweets.