ABSTRACT
While efforts have been made on bridging the semantic gap in image understanding, the in situ understanding of social media images is arguably more important but has had less progress. In this work, we enrich the representation of images in image tweets by considering their social context. We argue that in the microblog context, traditional image features, e.g., low-level SIFT or high-level detected objects, are far from adequate in interpreting the necessary semantics latent in image tweets. To bridge this gap, we move from the images' pixels to their context and propose a context-aware image bf tweet modelling (CITING) framework to mine and fuse contextual text to model such social media images' semantics. We start with tweet's intrinsic contexts, namely, 1) text within the image itself and 2) its accompanying text; and then we turn to the extrinsic contexts: 3) the external web page linked to by the tweet's embedded URL, and 4) the Web as a whole. These contexts can be leveraged to benefit many fundamental applications. To demonstrate the effectiveness our framework, we focus on the task of personalized image tweet recommendation, developing a feature-aware matrix factorization framework that encodes the contexts as a part of user interest modelling. Extensive experiments on a large Twitter dataset show that our proposed method significantly improves performance. Finally, to spur future studies, we have released both the code of our recommendation model and our image tweet dataset.
- O. Alonso, C. C. Marshall, and M. Najork. Are Some Tweets More Interesting Than Others? #HardQuestion. In Proc. of HCIR, 2013. Google ScholarDigital Library
- J. Bian, Y. Yang, and T.-S. Chua. Multimedia Summarization for Trending Topics in Microblogs. In Proc. of CIKM, 2013. Google ScholarDigital Library
- J. Bian, Y. Yang, and T.-S. Chua. Predicting Trending Messages and Diffusion Participants in Microblogging Network. In Proc. of SIGIR, 2014. Google ScholarDigital Library
- H. Cai, Y. Yang, X. Li, and Z. Huang. What Are Popular: Exploring Twitter Features for Event Detection, Tracking and Visualization. In Proc. of MM, 2015. Google ScholarDigital Library
- V. Campos, A. Salvador, X. Giro-i Nieto, and B. Jou. Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction. In Proc. of ASM, 2015. Google ScholarDigital Library
- E. F. Can, H. Oktay, and R. Manmatha. Predicting Retweet Count Using Visual Cues. In Proc. of CIKM, 2013. Google ScholarDigital Library
- K. Chen, T. Chen, G. Zheng, O. Jin, E. Yao, and Y. Yu. Collaborative Personalized Tweet Recommendation. In Proc. of SIGIR, 2012. Google ScholarDigital Library
- T. Chen, D. Lu, M.-Y. Kan, and P. Cui. Understanding and Classifying Image Tweets. In Proc. of MM, 2013. Google ScholarDigital Library
- T. Chen, H. M. SalahEldeen, X. He, M.-Y. Kan, and D. Lu. VELDA: Relating an Image Tweet's Text and Images. In Proc. of AAAI, 2015. Google ScholarDigital Library
- W. Feng and J. Wang. Retweet or Not?: Personalized Tweet Re-ranking. In Proc. of WSDM, 2013. Google ScholarDigital Library
- R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. Large-scale Matrix Factorization with Distributed Stochastic Gradient Descent. In Proc. of SIGKDD, 2011. Google ScholarDigital Library
- X. He, T. Chen, M.-Y. Kan, and X. Chen. TriRank: Review-aware Explainable Recommendation by Modeling Aspects. In Proc. of CIKM, 2015. Google ScholarDigital Library
- X. He, H. Zhang, and M.-Y. Kan. Fast Matrix Factorization for Online Recommendation with Implicit Feedback. In Proc. of SIGIR, 2016. Google ScholarDigital Library
- L. Hong, O. Dan, and B. D. Davison. Predicting Popular Messages in Twitter. In Proc. of WWW, 2011. Google ScholarDigital Library
- L. Hong, A. S. Doumith, and B. D. Davison. Co-factorization Machines: Modeling User Interests and Predicting Individual Decisions in Twitter. In Proc. of WSDM, 2013. Google ScholarDigital Library
- K. Ishiguro, A. Kimura, and K. Takeuchi. Towards Automatic Image Understanding and Mining via Social Curation. In Proc. of ICDM, 2012. Google ScholarDigital Library
- Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proc. of MM, 2014. Google ScholarDigital Library
- C. Kohlschütter, P. Fankhauser, and W. Nejdl. Boilerplate Detection Using Shallow Text Features. In Proc. of WSDM, 2010. Google ScholarDigital Library
- Y. Koren, R. Bell, and C. Volinsky. Matrix Factorization Techniques for Recommender Systems. Computer, 42(8):30--37, 2009. Google ScholarDigital Library
- H. Kwak, C. Lee, H. Park, and S. Moon. What is Twitter, a Social Network or a News Media? In Proc. of WWW, 2010. Google ScholarDigital Library
- D. Laniado and P. Mika. Making Sense of Twitter. In Proc. of ISWC, 2010. Google ScholarDigital Library
- N. Naveed, T. Gottron, J. Kunegis, and A. C. Alhadi. Bad News Travel Fast: A Content-based Analysis of Interestingness on Twitter. In Proc. of WebSci, 2011. Google ScholarDigital Library
- S. Petrovic, M. Osborne, and V. Lavrenko. RT to Win! Predicting Message Propagation in Twitter. In Proc. of ICWSM, 2011.Google Scholar
- S. Rendle. Factorization Machines. In Proc. of ICDM, 2010. Google ScholarDigital Library
- S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proc. of UAI, 2009. Google ScholarDigital Library
- M. Schinas, S. Papadopoulos, Y. Kompatsiaris, and P. A. Mitkas. Visual Event Summarization on Social Media Using Topic Modelling and Graph-based Ranking Algorithms. In Proc. of ICMR, 2015. Google ScholarDigital Library
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going Deeper with Convolutions. In Proc. of CVPR, 2015.Google ScholarCross Ref
- I. Uysal and W. B. Croft. User Oriented Tweet Ranking: A Filtering Approach to Microblogs. In Proc. of CIKM, 2011. Google ScholarDigital Library
- A. Wang, T. Chen, and M.-Y. Kan. Re-tweeting from a Linguistic Perspective. In Proc. of LSM, 2012. Google ScholarDigital Library
- Z. Wang, P. Cui, L. Xie, H. Chen, W. Zhu, and S. Yang. Analyzing Social Media via Event Facets. In Proc. of MM, 2012. Google ScholarDigital Library
- Z. Wang, P. Cui, L. Xie, W. Zhu, Y. Rui, and S. Yang. Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs. ACM TOMM, 10(4):34:1--34:21, 2014. Google ScholarDigital Library
- R. Yan, M. Lapata, and X. Li. Tweet Recommendation with Graph Co-ranking. In Proc. of ACL, 2012. Google ScholarDigital Library
- T. R. Zaman, R. Herbrich, J. V. Gael, and D. Stern. Predicting Information Spreading in Twitter. In Computational Social Science and the Wisdom of Crowds Workshop, 2010.Google Scholar
- H. Zhang, G. Kim, and E. P. Xing. Dynamic Topic Modeling for Monitoring Market Competition from Online Text and Image Data. In Proc. of KDD, 2015. Google ScholarDigital Library
- H. Zhang, X. Shang, H. Luan, M. Wang, and T.-S. Chua. Learning from Collective Intelligence: Feature Learning Using Social Images and Tags. ACM TOMM, 2016.Google ScholarDigital Library
Index Terms
- Context-aware Image Tweet Modelling and Recommendation
Recommendations
Followee recommendation based on topic extraction and sentiment analysis from tweets
iiWAS '15: Proceedings of the 17th International Conference on Information Integration and Web-based Applications & ServicesTwitter has become a popular social media service, accumulating and distributing vast amounts of information for its numerous users. One feature of Twitter is that it enables a user to follow other users, who can obtain the information her/his followees ...
Collaborative personalized tweet recommendation
SIGIR '12: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrievalTwitter has rapidly grown to a popular social network in recent years and provides a large number of real-time messages for users. Tweets are presented in chronological order and users scan the followees' timelines to find what they are interested in. ...
User-Oriented Context Suggestion
UMAP '16: Proceedings of the 2016 Conference on User Modeling Adaptation and PersonalizationRecommender systems have been used in many domains to assist users' decision making by providing item recommendations and thereby reducing information overload. Context-aware recommender systems go further, incorporating the variability of users' ...
Comments