Ekraam Sabir
ABSTRACT
Nefarious actors on social media and other platforms often spread rumors and falsehoods through images whose metadata (e.g., captions) have been modified to provide visual substantiation of the rumor/falsehood. This type of modification is referred to as image repurposing, in which often an unmanipulated image is published along with incorrect or manipulated metadata to serve the actor's ulterior motives. We present the Multimodal Entity Image Repurposing (MEIR) dataset, a substantially challenging dataset over that which has been previously available to support research into image repurposing detection. The new dataset includes location, person, and organization manipulations on real-world data sourced from Flickr. We also present a novel, end-to-end, deep multimodal learning model for assessing the integrity of an image by combining information extracted from the image with related information from a knowledge base. The proposed method is compared against state-of-the-art techniques on existing datasets as well as MEIR, where it outperforms existing methods across the board, with AUC improvement up to 0.23.
- Khurshid Asghar, Zulfiqar Habib, and Muhammad Hussain. 2017. Copy-move and splicing image forgery detection and localization techniques: a review. Australian Journal of Forensic Sciences , Vol. 49, 3 (2017), 281--307.Google Scholar
Cross Ref
- Ella Bingham and Heikki Mannila. 2001. Random Projection in Dimensionality Reduction: Applications to Image and Text Data. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '01). ACM, New York, NY, USA, 245--250. Google ScholarDigital Library
- Christina Boididou, Katerina Andreadou, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, Michael Riegler, and Yiannis Kompatsiaris. 2015. Verifying Multimedia Use at MediaEval 2015.. In MediaEval.Google Scholar
- Leo Breiman. 2001. Random Forests . Machine Learning , Vol. 45, 1 (Oct. 2001), 5--32. Google ScholarDigital Library
- Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and regression trees. http://cds.cern.ch/record/2253780Google Scholar
- Sanjoy Dasgupta. 2000. Experiments with random projection. In Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 143--151. Google ScholarDigital Library
- Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting on association for computational linguistics. Association for Computational Linguistics, 363--370. Google ScholarDigital Library
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation , Vol. 9, 8 (1997), 1735--1780. Google ScholarDigital Library
- Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. 2017. Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 1465--1471. Google ScholarDigital Library
- Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017a. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 795--816. Google ScholarDigital Library
- Zhiwei Jin, Juan Cao, Yazi Zhang, and Yongdong Zhang. 2015. MCG-ICT at MediaEval 2015: Verifying Multimedia Use with a Two-Level Classification Model.. In MediaEval.Google Scholar
- Z. Jin, J. Cao, Y. Zhang, J. Zhou, and Q. Tian. 2017b. Novel Visual and Statistical Image Features for Microblogs News Verification . IEEE Transactions on Multimedia , Vol. 19, 3 (March 2017), 598--608. Google ScholarDigital Library
- J. R. Kettenring. 1971. Canonical analysis of several sets of variables. Biometrika , Vol. 58, 3 (Dec. 1971), 433--451.Google ScholarCross Ref
- Regina Marchi. 2012. With Facebook, Blogs, and Fake News, Teens Reject Journalistic "Objectivity". Journal of Communication Inquiry , Vol. 36, 3 (July 2012), 246--262.Google ScholarCross Ref
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality . In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3111--3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf Google ScholarDigital Library
- Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python . Journal of Machine Learning Research , Vol. 12 (Oct. 2011), 2825--2830. http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html Google ScholarDigital Library
- Muhammad Ali Qureshi and Mohamed Deriche. 2015. A bibliography of pixel-based blind image forgery detection techniques. Signal Processing: Image Communication , Vol. 39 (2015), 46--74. http://www.sciencedirect.com/science/article/pii/S0923596515001393 Google ScholarDigital Library
- Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. CSI: A Hybrid Deep Model for Fake News Detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 797--806. Google ScholarDigital Library
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision , Vol. 115, 3 (Dec. 2015), 211--252. Google ScholarDigital Library
- Cencheng Shen, Ming Sun, Minh Tang, and Carey E. Priebe. 2014. Generalized canonical correlation analysis for classification. Journal of Multivariate Analysis , Vol. 130, C (2014), 310--322. https://econpapers.repec.org/article/eeejmvana/v_3a130_3ay_3a2014_3ai_3ac_3ap_3a310--322.htm Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition . arXiv:1409.1556 {cs} (Sept. 2014). http://arxiv.org/abs/1409.1556 arXiv: 1409.1556.Google Scholar
- Ratnam Singh and Mandeep Kaur. 2016. Copy Move Tampering Detection Techniques: A Review . International Journal of Applied Engineering Research , Vol. 11, 5 (2016), 3610--3615. https://www.ripublication.com/ijaer16/ijaerv11n5_104.pdfGoogle Scholar
- Ming Sun, Carey E. Priebe, and Minh Tang. 2013. Generalized canonical correlation analysis for disparate data fusion. Pattern Recognition Letters , Vol. 34, 2 (2013), 194--200. Google ScholarDigital Library
- Yue Wu, Wael Abd-Almageed, and Prem Natarajan. 2017. Deep Matching and Validation Network: An End-to-End Solution to Constrained Image Splicing Localization and Detection. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 1480--1502. Google ScholarDigital Library
- Markos Zampoglou, Symeon Papadopoulos, Yiannis Kompatsiaris, Ruben Bouwmeester, and Jochen Spangenberg. 2016. Web and Social Media Image Forensics for News Professionals.. In SMN@ ICWSM. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/download/13206/12860Google Scholar
Index Terms
- Deep Multimodal Image-Repurposing Detection
Recommendations
Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs
MM '17: Proceedings of the 25th ACM international conference on MultimediaMicroblogs have become popular media for news propagation in recent years. Meanwhile, numerous rumors and fake news also bloom and spread wildly on the open social media platforms. Without verification, they could seriously jeopardize the credibility of ...
Rumor detection in social networks via deep contextual modeling
ASONAM '19: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and MiningFake news and rumors constitute a major problem in social networks recently. Due to the fast information propagation in social networks, it is inefficient to use human labor to detect suspicious news. Automatic rumor detection is thus necessary to ...
VRoC: Variational Autoencoder-aided Multi-task Rumor Classifier Based on Text
WWW '20: Proceedings of The Web Conference 2020Social media became popular and percolated almost all aspects of our daily lives. While online posting proves very convenient for individual users, it also fosters fast-spreading of various rumors. The rapid and wide percolation of rumors can cause ...
Comments