Wyverson Bonasoli de Oliveira
Thiago H. Silva
Skip Abstract Section
Abstract
Opinion mining in outdoor images posted by users during different activities can provide valuable information to better understand urban areas. In this regard, we propose a framework to classify the sentiment of outdoor images shared by users on social networks. We compare the performance of state-of-the-art ConvNet architectures and one specifically designed for sentiment analysis. We also evaluate how the merging of deep features and semantic information derived from the scene attributes can improve classification and cross-dataset generalization performance. The evaluation explores a novel dataset—namely, OutdoorSent—and other publicly available datasets. We observe that the incorporation of knowledge about semantic attributes improves the accuracy of all ConvNet architectures studied. Besides, we found that exploring only images related to the context of the study—outdoor, in our case—is recommended, i.e., indoor images were not significantly helpful. Furthermore, we demonstrated the applicability of our results in the United States city of Chicago, Illinois, showing that they can help to improve the knowledge of subjective characteristics of different areas of the city. For instance, particular areas of the city tend to concentrate more images of a specific class of sentiment, which are also correlated with median income, opening up opportunities in different fields.
- Unaiza Ahsan, Munmun De Choudhury, and Irfan Essa. 2017. Towards using visual attributes to infer image sentiment of social events. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 1372--1379. DOI:https://doi.org/10.1109/IJCNN.2017.7966013Google Scholar
Cross Ref
- Oscar Araque, Ignacio Corcuera-Platas, J. Fernando Sánchez-Rada, and Carlos A. Iglesias. 2017. Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Exp. Syst. Applic. 77 (2017), 236--246. DOI:https://doi.org/10.1016/j.eswa.2017.02.002Google ScholarDigital Library
- Ramy Baly, Roula Hobeica, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, and Ahmad Al-Sallab. 2016. A meta-framework for modeling the human reading process in sentiment analysis. ACM Trans. Inf. Syst. 35, 1, Article 7 (Aug. 2016), 21 pages. DOI:https://doi.org/10.1145/2950050Google ScholarDigital Library
- Fabrício Benevenuto, Matheus Araújo, and Filipe Ribeiro. 2015. Sentiment analysis methods for social media. In Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia’15). ACM, 11--11. DOI:https://doi.org/10.1145/2820426.2820642Google ScholarDigital Library
- Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the ACM International Conference on Multimedia. ACM, 223--232. DOI:https://doi.org/10.1145/2502081.2502282Google ScholarDigital Library
- Guoyong Cai and Binbin Xia. 2015. Convolutional neural networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Computing. Springer International Publishing, Cham, 159--167. DOI:https://doi.org/10.1007/978-3-319-25207-0_14Google Scholar
- Víctor Campos, Brendan Jou, and Xavier Giró i Nieto. 2017. From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction. Image Vis. Comput. 65 (2017), 15--22. DOI:https://doi.org/10.1016/j.imavis.2017.01.011Google ScholarDigital Library
- Fuhai Chen, Rongrong Ji, Jinsong Su, Donglin Cao, and Yue Gao. 2018. Predicting microblog sentiments via weakly supervised multimodal deep learning. IEEE Trans. Multim. 20, 4 (Apr. 2018), 997--1007. DOI:https://doi.org/10.1109/TMM.2017.2757769Google ScholarDigital Library
- Tao Chen, Damian Borth, Trevor Darrell, and Shih-Fu Chang. 2014. DeepSentiBank: Visual sentiment concept classification with deep convolutional neural networks. CoRR abs/1410.8586 (2014), 1--7.Google Scholar
- Yu-Hsiu Chen, Ting-Hsuan Chao, Sheng-Yi Bai, Yen-Liang Lin, Wen-Chin Chen, and Winston H. Hsu. 2015. Filter-invariant image classification on social media photos. In Proceedings of the ACM International Conference on Multimedia (MM’15). ACM, 855--858. DOI:https://doi.org/10.1145/2733373.2806348Google Scholar
- Justin Cranshaw, Raz Schwartz, Jason I. Hong, and Norman M. Sadeh. 2012. The livehoods project: Utilizing social media to understand the dynamics of a city. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--8.Google Scholar
- CrowdFlower. 2015. Image Sentiment Polarity Classification (Dataset). CrowdFlower. Retrieved from https://data.world/crowdflower/image-sentiment-polarity.Google Scholar
- Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--10.Google Scholar
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255. DOI: https://doi.org/10.1109/CVPR.2009.5206848Google ScholarCross Ref
- Abhinav Dhall, Roland Goecke, and Tom Gedeon. 2015. Automatic group happiness intensity analysis. IEEE Trans. Affect. Comput. 6, 1 (Jan. 2015), 13--26. DOI:https://doi.org/10.1109/TAFFC.2015.2397456Google ScholarCross Ref
- Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the International Conference on Knowledge Discovery and Data Mining. AAAI Press, 226--231. Retrieved from http://dl.acm.org/citation.cfm?id=3001460.3001507.Google Scholar
- Fuhai Chen, Yue Gao, Donglin Cao, and Rongrong Ji. 2015. Multimodal hypergraph learning for microblog sentiment prediction. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’15). IEEE, 1--6. DOI:https://doi.org/10.1109/ICME.2015.7177477Google Scholar
- Scott A. Golder and Michael W. Macy. 2011. Diurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures. Science 333, 6051 (2011), 1878--1881. DOI:https://doi.org/10.1126/science.1202775Google ScholarCross Ref
- Alan Hanjalic, Christoph Kofler, and Martha Larson. 2012. Intent and its discontents: The user at the wheel of the online video search engine. In Proceedings of the ACM International Conference on Multimedia. ACM, 1239--1248. DOI:https://doi.org/10.1145/2393347.2396424Google ScholarDigital Library
- K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 770--778. DOI:https://doi.org/10.1109/CVPR.2016.90Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735--1780. DOI:https://doi.org/10.1162/neco.1997.9.8.1735Google ScholarDigital Library
- Steven Hoffman, Renu Sharma, and Arun Ross. 2018. Convolutional neural networks for iris presentation attack detection: Toward cross-dataset and cross-sensor generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18) Workshops. IEEE, 1620--1628. DOI:https://doi.org/10.1109/CVPRW.2018.00213Google ScholarCross Ref
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 4700--4708. DOI:https://doi.org/10.1109/CVPR.2017.243Google Scholar
- Minlie Huang, Qiao Qian, and Xiaoyan Zhu. 2017. Encoding syntactic knowledge in neural networks for sentiment classification. ACM Trans. Inf. Syst. 35, 3, Article 26 (June 2017), 27 pages. DOI:https://doi.org/10.1145/3052770Google ScholarDigital Library
- Rongrong Ji, Donglin Cao, Yiyi Zhou, and Fuhai Chen. 2016. Survey of visual sentiment prediction for social media analysis. Front. Comput. Sci. 10, 4 (01 Aug. 2016), 602--611. DOI:https://doi.org/10.1007/s11704-016-5453-2Google Scholar
- Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, and Jie Tang. 2012. Can we understand Van Gogh’s mood?: Learning to infer affects from images in social networks. In Proceedings of the ACM International Conference on Multimedia. ACM, 857--860. DOI:https://doi.org/10.1145/2393347.2396330Google ScholarDigital Library
- M. Katsurai and S. Satoh. 2016. Image sentiment analysis using latent correlations among visual, textual, and sentiment views. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’16). IEEE, 2837--2841. DOI:https://doi.org/10.1109/ICASSP.2016.7472195Google Scholar
- George L. Kelling and Catherine M. Coles. 1997. Fixing Broken Windows: Restoring Order and Reducing Crime in Our Communities. Simon and Schuster, New York.Google Scholar
- Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proc. Nat. Acad. Sci. 111, 24 (2014), 8788--8790.Google ScholarCross Ref
- Géraud Le Falher, Aristides Gionis, and Michael Mathioudakis. 2015. Where is the Soho of Rome? Measures and algorithms for finding similar neighborhoods in cities. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--23.Google Scholar
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (May 2015), 436--444. Retrieved from http://dx.doi.org/10.1038/nature14539.Google ScholarCross Ref
- Bing Li, Songhe Feng, Weihua Xiong, and Weiming Hu. 2012. Scaring or pleasing: Exploit emotional impact of an image. In Proceedings of the ACM International Conference on Multimedia. ACM, 1365--1366.Google ScholarDigital Library
- Dazhen Lin, Lingxiao Li, Donglin Cao, Yanping Lv, and Xiao Ke. 2018. Multi-modality weakly labeled sentiment learning based on explicit emotion signal for Chinese microblog. Neurocomputing 272 (2018), 258--269. DOI: https://doi.org/10.1016/j.neucom.2017.06.078Google ScholarDigital Library
- Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia. ACM, 83--92. DOI:https://doi.org/10.1145/1873951.1873965Google ScholarDigital Library
- Nicolas Maisonneuve, Matthias Stevens, Maria E. Niessen, and Luc Steels. 2009. NoiseTube: Measuring and mapping noise pollution with mobile phones. In Information Technologies in Environmental Engineering. Springer, 215--228. DOI:https://doi.org/10.1007/978-3-540-88351-7_16Google Scholar
- Stuart E. Middleton and Vadims Krivcovs. 2016. Geoparsing and geosemantics for social media: Spatiotemporal grounding of content propagating rumors to support trust and veracity analysis during breaking news. ACM Trans. Inf. Syst. 34, 3, Article 16 (Apr. 2016), 26 pages. DOI:https://doi.org/10.1145/2842604Google ScholarDigital Library
- Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH’10), Takao Kobayashi, Keikichi Hirose, and Satoshi Nakamura (Eds.). ISCA, 1045--1048. Retrieved from http://dblp.uni-trier.de/db/conf/interspeech/interspeech2010.html#MikolovKBCK10.Google ScholarCross Ref
- Willi Mueller, Thiago H. Silva, Jussara M. Almeida, and Antonio A. F. Loureiro. 2017. Gender matters! Analyzing global cultural gender preferences for venues using social sensing. EPJ Data Sci. 6, 1 (2017), 5. DOI:https://doi.org/10.1140/epjds/s13688-017-0101-0Google ScholarCross Ref
- Alessandro Ortis, Giovanni M. Farinella, Giovanni Torrisi, and Sebastiano Battiato. 2018. Visual sentiment analysis based on objective text description of images. In Proceedings of the International Conference on Content-based Multimedia Indexing (CBMI’18). IEEE, 1--6. DOI:https://doi.org/10.1109/CBMI.2018.8516481Google ScholarCross Ref
- George Parrett. 2016. 3.5 Million Photos Shared Every Minute in 2016. Deloitte. Retrieved from https://goo.gl/uwF81P.Google Scholar
- Genevieve Patterson and James Hays. 2012. SUN attribute database: Discovering, annotating, and recognizing scene attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 2751--2758. DOI:https://doi.org/10.1109/CVPR.2012.6247998Google ScholarDigital Library
- Genevieve Patterson, Chen Xu, Hang Su, and James Hays. 2014. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis. 108, 1--2 (2014), 59--81. DOI:https://doi.org/10.1007/s11263-013-0695-zGoogle ScholarDigital Library
- Soujanya Poria, Haiyun Peng, Amir Hussain, Newton Howard, and Erik Cambria. 2017. Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261 (2017), 217--230. DOI:https://doi.org/10.1016/j.neucom.2016.09.117Google ScholarCross Ref
- Daniele Quercia, Rossano Schifanella, and Luca Maria Aiello. 2014. The shortest path to happiness: Recommending beautiful, quiet, and happy routes in the city. In Proceedings of the ACM Conference on Hypertext and Social Media. ACM, 116--125. DOI:https://doi.org/10.1145/2631775.2631799Google ScholarDigital Library
- Daniele Quercia, Rossano Schifanella, Luca Maria Aiello, and Kate McLean. 2015. Smelly maps: The digital life of urban smellscapes. In Proceedings of the International Conference on Web and Social Media (ICWSM’15), Vol. 1. AAAI Press, 327--336.Google Scholar
- Joseph Redmon and Ali Farhadi. 2017. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 7263--7271. DOI:https://doi.org/10.1109/CVPR.2017.690Google ScholarCross Ref
- Filipe N. Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. 2016. SentiBench—A benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Sci. 5, 1 (2016), 1--29. DOI:https://doi.org/10.1140/epjds/s13688-016-0085-1Google ScholarCross Ref
- Darshan Santani, Salvador Ruiz-Correa, and Daniel Gatica-Perez. 2018. Looking south: Learning urban perception in developing cities. ACM Trans. Soc. Comput. 1, 3, Article 13 (Dec. 2018), 23 pages. DOI:https://doi.org/10.1145/3224182Google ScholarDigital Library
- Frances A. Santos, Thiago H. Silva, Antonio A. F. Loureiro, and Leandro A. Villas. 2018. Uncovering the perception of urban outdoor areas expressed in social media. In Proceedings of the International Conference on Web Intelligence (WI’18), Vol. 1. IEEE/WIC/ACM, 120--127. DOI:https://doi.org/10.1109/WI.2018.00-99Google Scholar
- Stefan Siersdorfer, Enrico Minack, Fan Deng, and Jonathon Hare. 2010. Analyzing and predicting sentiment of images on the social web. In Proceedings of the ACM International Conference on Multimedia. ACM, 715--718. DOI: https://doi.org/10.1145/1873951.1874060Google ScholarDigital Library
- Thiago H. Silva, Pedro O. S. Vaz de Melo, Jussara M. Almeida, Mirco Musolesi, and Antonio A. F. Loureiro. 2017. A large-scale study of cultural differences using urban data about eating and drinking preferences. Inf. Syst. 72, Supplement C (2017), 95--116. DOI:https://doi.org/10.1016/j.is.2017.10.002Google ScholarCross Ref
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014), 1--14.Google Scholar
- Kaikai Song, Ting Yao, Qiang Ling, and Tao Mei. 2018. Boosting image sentiment analysis with visual attention. Neurocomputing 312 (2018), 218--228. DOI:https://doi.org/10.1016/j.neucom.2018.05.104Google ScholarDigital Library
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 2818--2826. DOI:https://doi.org/10.1109/CVPR.2016.308Google ScholarCross Ref
- Antonio Torralba and Alexei A. Efros. 2011. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE Computer Society, 1521--1528. DOI:https://doi.org/10.1109/CVPR.2011.5995347Google Scholar
- Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. 2017. Cross-media learning for image sentiment analysis in the wild. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW’17), Vol. 1. IEEE, 308--317. DOI:https://doi.org/10.1109/ICCVW.2017.45Google ScholarCross Ref
- Jianlong Wu, Zhouchen Lin, and Hongbin Zha. 2016. Multi-view common space learning for emotion recognition in the wild. In Proceedings of the ACM International Conference on Multimodal Interaction. ACM, 464--471. DOI: https://doi.org/10.1145/2993148.2997631Google ScholarDigital Library
- Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2015. Joint visual-textual sentiment analysis with deep neural networks. In Proceedings of the ACM International Conference on Multimedia (MM’15). ACM, 1071--1074. DOI: https://doi.org/10.1145/2733373.2806284Google ScholarDigital Library
- Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2015. Robust image sentiment analysis using progressively trained and domain transferred deep networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’15). AAAI Press, 381--388. Retrieved from http://dl.acm.org/citation.cfm?id=2887007.2887061.Google Scholar
- Jianbo Yuan, Sean Mcdonough, Quanzeng You, and Jiebo Luo. 2013. Sentribute: Image sentiment analysis from a mid-level perspective. In Proceedings of the International Workshop on Issues of Sentiment Discovery and Opinion Mining. ACM, Article 10, 8 pages. DOI:https://doi.org/10.1145/2502069.2502079Google ScholarDigital Library
- Xueying Zhan, Yaowei Wang, Yanghui Rao, and Qing Li. 2019. Learning from multi-annotator data: A noise-aware classification framework. ACM Trans. Inf. Syst. 37, 2, Article 26 (Feb. 2019), 28 pages. DOI:https://doi.org/10.1145/3309543Google ScholarDigital Library
- Liang Zhao, Feng Chen, Chang-Tien Lu, and Naren Ramakrishnan. 2016. Online spatial event forecasting in microblogs. ACM Trans. Spatial Algor. Syst. 2, 4, Article 15 (Nov. 2016), 39 pages. DOI:https://doi.org/10.1145/2997642Google Scholar
- Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40, 6 (June 2018), 1452--1464. DOI:https://doi.org/10.1109/TPAMI.2017.2723009Google ScholarCross Ref
- Guang-You Zhou and Jimmy Xiangji Huang. 2017. Modeling and mining domain shared knowledge for sentiment analysis. ACM Trans. Inf. Syst. 36, 2, Article 18 (Aug. 2017), 36 pages. DOI:https://doi.org/10.1145/3091995Google ScholarDigital Library
Index Terms
- OutdoorSent: Sentiment Analysis of Urban Outdoor Images by Using Semantic and Deep Features
Recommendations
Does sentiment help requirement engineering: exploring sentiments in user comments to discover informative comments
AbstractUser comments are valuable resources for software improvement; however, owing to excessive volume, informative comments need to be selectively analyzed. We attempt to address this problem by sentiment analysis and expect sentiment can be a useful ...
Sentiment Analysis in Outdoor Images Using Deep Learning
WebMedia '18: Proceedings of the 24th Brazilian Symposium on Multimedia and the WebIn this work, we explore how Convolutional Neural Networks can be applied to the task of sentiment analysis in visual media. We compare four different architectures and propose a new approach where attributes that represent the main categories used for ...
A hybrid deep learning architecture for opinion-oriented multi-document summarization based on multi-feature fusion
AbstractOpinion summarization is a process to produce concise summaries from a large number of opinionated texts. In this paper, we present a novel deep-learning-based method for the generic opinion-oriented extractive summarization of multi-...
Highlights- A novel deep-learning-based method for opinion-oriented multi-document summarization.
Comments