ABSTRACT
A number of online services nowadays rely upon machine learning to extract valuable information from data collected in the wild. This exposes learning algorithms to the threat of data poisoning, i.e., a coordinate attack in which a fraction of the training data is controlled by the attacker and manipulated to subvert the learning process. To date, these attacks have been devised only against a limited class of binary learning algorithms, due to the inherent complexity of the gradient-based procedure used to optimize the poisoning points (a.k.a. adversarial training examples). In this work, we first extend the definition of poisoning attacks to multiclass problems. We then propose a novel poisoning algorithm based on the idea of back-gradient optimization, i.e., to compute the gradient of interest through automatic differentiation, while also reversing the learning procedure to drastically reduce the attack complexity. Compared to current poisoning strategies, our approach is able to target a wider class of learning algorithms, trained with gradient-based procedures, including neural networks and deep learning architectures. We empirically evaluate its effectiveness on several application examples, including spam filtering, malware detection, and handwritten digit recognition. We finally show that, similarly to adversarial test examples, adversarial training examples can also be transferred across different learning algorithms.
- Marco Barreno, Blaine Nelson, Anthony Joseph, and J. Tygar. 2010. The security of machine learning. Machine Learning 81 (2010), 121--148. Issue 2.Google ScholarDigital Library
- Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. D. Tygar. 2006. Can machine learning be secure? In Proc. ACM Symp. Information, Computer and Comm. Sec. (ASIACCS '06). ACM, New York, NY, USA, 16--25. Google ScholarDigital Library
- Y. Bengio. 2000. Gradient-based optimization of hyperparameters. Neural Computation 12, 8 (2000), 1889--1900. Google ScholarDigital Library
- Battista Biggio, Samuel Rota Bulò, Ignazio Pillai, Michele Mura, Eyasu Zemene Mequanint, Marcello Pelillo, and Fabio Roli. 2014. Poisoning complete-linkage hierarchical clustering. In Joint IAPR Int'l Workshop on Structural, Syntactic, and Statistical Pattern Recognition (Lecture Notes in Computer Science), P. Franti, G. Brown, M. Loog, F. Escolano, and M. Pelillo (Eds.), Vol. 8621. Springer Berlin Heidelberg, Joensuu, Finland, 42--52. Google ScholarDigital Library
- Battista Biggio, Igino Corona, Giorgio Fumera, Giorgio Giacinto, and Fabio Roli. 2011. Bagging Classifiers for Fighting Poisoning Attacks in Adversarial Classification Tasks. In 10th International Workshop on Multiple Classifier Systems (MCS) (Lecture Notes in Computer Science), Carlo Sansone, Josef Kittler, and Fabio Roli (Eds.), Vol. 6713. Springer-Verlag, 350--359. Google ScholarCross Ref
- B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndi, P. Laskov, G. Giacinto, and F. Roli. 2013. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), Part III (LNCS), Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Železný (Eds.), Vol. 8190. Springer Berlin Heidelberg, 387--402. Google ScholarDigital Library
- Battista Biggio, Giorgio Fumera, and Fabio Roli. 2014. Security Evaluation of Pattern Classifiers Under Attack. IEEE Transactions on Knowledge and Data Engineering 26, 4 (April 2014), 984--996. Google ScholarDigital Library
- Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines, In 29th Int'l Conf. on Machine Learning, John Langford and Joelle Pineau (Eds.). Int'l Conf. on Machine Learning (ICML), 1807--1814.Google ScholarDigital Library
- Battista Biggio, Ignazio Pillai, Samuel Rota Bulò, Davide Ariu, Marcello Pelillo, and Fabio Roli. 2013. Is Data Clustering in Adversarial Settings Secure? In Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security (AISec '13). ACM, New York, NY, USA, 87--98. Google ScholarDigital Library
- Battista Biggio, Konrad Rieck, Davide Ariu, Christian Wressnegger, Igino Corona, Giorgio Giacinto, and Fabio Roli. 2014. Poisoning Behavioral Malware Clustering. In 2014 Workshop on Artificial Intelligent and Security (AISec '14). ACM, New York, NY, USA, 27--36. Google ScholarDigital Library
- C. Blake and C. J. Merz. 1998. UCI Repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html (1998).Google Scholar
- Nader H. Bshouty, Nadav Eiron, and Eyal Kushilevitz. 1999. PAC Learning with Nasty Noise. In Algorithmic Learning Theory, Osamu Watanabe and Takashi Yokomori (Eds.). Lecture Notes in Computer Science, Vol. 1720. Springer Berlin Heidelberg, 206--218. https://doi.org/10.1007/3-540-46769-6_17Google Scholar
- C. Do, C. S. Foo, and A. Y. Ng. 2008. Efficient multiple hyperparameter learning for log-linear models. In Advances in Neural Information Processing Systems. 377--384.Google Scholar
- Justin Domke. 2012. Generic Methods for Optimization-Based Modeling. In 15th Int'l Conf. Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Neil D. Lawrence and Mark Girolami (Eds.), Vol. 22. PMLR, La Palma, Canary Islands, 318--326.Google Scholar
- Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations.Google Scholar
- L. Huang, A. D. Joseph, B. Nelson, B. Rubinstein, and J. D. Tygar. 2011. Adversarial Machine Learning. In 4th ACM Workshop on Artificial Intelligence and Security (AISec 2011). Chicago, IL, USA, 43--57. Google ScholarDigital Library
- Anthony D. Joseph, Pavel Laskov, Fabio Roli, J. Doug Tygar, and Blaine Nelson. 2013. Machine Learning Methods for Computer Security (Dagstuhl Perspectives Workshop 12371). Dagstuhl Manifestos 3, 1 (2013), 1--30.Google Scholar
- Michael Kearns and Ming Li. 1993. Learning in the presence of malicious errors. SIAM J. Comput. 22, 4 (1993), 807--837. https://doi.org/10.1137/0222052 Google ScholarDigital Library
- Marius Kloft and Pavel Laskov. 2012. Security Analysis of Online Centroid Anomaly Detection. Journal of Machine Learning Research 13 (2012), 3647--3690.Google ScholarDigital Library
- P. W. Koh and P. Liang. 2017. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning (ICML).Google Scholar
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. GradientBased Learning Applied to Document Recognition. In Proceedings of the IEEE, Vol. 86. 2278--2324. Google ScholarCross Ref
- Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. 2015. Gradient-based Hyperparameter Optimization Through Reversible Learning. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 2113--2122.Google Scholar
- Shike Mei and Xiaojin Zhu. 2015. Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. In 29th AAAI Conf. Artificial Intelligence (AAAI '15).Google ScholarDigital Library
- Seyed-Mohsen, Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In CVPR.Google Scholar
- B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. P. Rubinstein, U. Saini, C.A. Sutton, J. D. Tygar, and K. Xia. 2008. Exploiting Machine Learning to Subvert your Spam Filter. LEET 8 (2008), 1--9.Google ScholarDigital Library
- Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I. P. Rubinstein, Udam Saini, Charles Sutton, J. D. Tygar, and Kai Xia. 2008. Exploiting machine learning to subvert your spam filter. In LEET'08: Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats. USENIX Association, Berkeley, CA, USA, 1--9.Google ScholarDigital Library
- Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks Against Machine Learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (ASIA CCS '17). ACM, New York, NY, USA, 506--519. Google ScholarDigital Library
- Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. 2016. The Limitations of Deep Learning in Adversarial Settings. In Proc. 1st IEEE European Symposium on Security and Privacy. IEEE, 372--387. Google ScholarCross Ref
- K. R. Patil, X. Zhu, L. Kope?, and B. C. Love. 2014. Optimal teaching for limitedcapacity human learners. In Advances in Neural Information Processing Systems. 2465--2473.Google Scholar
- B. A. Pearlmutter. 1994. Fast Exact Multiplication by the Hessian. Neural Computation 6, 1 (1994), 147--160. Google ScholarDigital Library
- F. Pedregosa. 2016. Hyperparameter optimization with approximate gradient. In 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research), Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. PMLR, New York, New York, USA, 737--746.Google Scholar
- Benjamin I. P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shinghon Lau, Satish Rao, Nina Taft, and J. D. Tygar. 2009. ANTIDOTE: understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Internet Measurement Conference (IMC '09). ACM, New York, NY, USA, 1--14. Google ScholarDigital Library
- D. Sgandurra, L. Muñoz-González, R. Mohsen, and E. C. Lupu. 2016. Automated Dynamic Analysis of Ransomware: Benefits, Limitations and use for Detection. arXiv preprint arXiv:1609.03020 (2016).Google Scholar
- Charles Smutz and Angelos Stavrou. 2012. Malicious PDF Detection Using Metadata and Structural Features. In Proceedings of the 28th Annual Computer Security Applications Conference (ACSAC '12). ACM, New York, NY, USA, 239--248. Google ScholarDigital Library
- J. Steinhardt, P. W. Koh, and P. Liang. 2017. Certified Defenses for Data Poisoning Attacks. arXiv preprint arXiv:1706.03691 (2017).Google Scholar
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. http://arxiv.org/abs/1312.6199Google Scholar
- Nedim Šrndic and Pavel Laskov. 2014. Practical Evasion of a Learning-Based Classifier: A Case Study. In Proc. 2014 IEEE Symp. Security and Privacy (SP '14). IEEE CS, Washington, DC, USA, 197--211.Google ScholarDigital Library
- Gang Wang, Tianyi Wang, Haitao Zheng, and Ben Y. Zhao. 2014. Man vs. Machine: Practical Adversarial Detection of Malicious Crowdsourcing Workers. In 23rd USENIX Security Symposium (USENIX Security 14). USENIX Association, San Diego, CA.Google Scholar
- Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. 2015. Is Feature Selection Secure against Training Data Poisoning? In JMLR W&CP - Proc. 32nd Int'l Conf. Mach. Learning (ICML), Francis Bach and David Blei (Eds.), Vol. 37. 1689--1698.Google Scholar
- X. Zhu. 2013. Machine Teaching for Bayesian Learners in the Exponential Family. In Advances in Neural Information Processing Systems. 1905--1913.Google Scholar
Index Terms
- Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Recommendations
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
CCS '18: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications SecurityDeep neural networks and machine-learning algorithms are pervasively used in several applications, ranging from computer vision to computer security. In most of these applications, the learning algorithm has to face intelligent and adaptive attackers ...
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
In recent years, machine learning algorithms, and more specifically deep learning algorithms, have been widely used in many fields, including cyber security. However, machine learning systems are vulnerable to adversarial attacks, and this limits the ...
Adversarial examples: A survey of attacks and defenses in deep learning-enabled cybersecurity systems
AbstractOver the last few years, the adoption of machine learning in a wide range of domains has been remarkable. Deep learning, in particular, has been extensively used to drive applications and services in specializations such as computer vision, ...
Highlights- A taxonomy of cybersecurity applications is established.
- Adversarial machine learning is systematically overviewed.
- An extensive, curated list of cybersecurity-related datasets is provided.
- Methods for generating adversarial ...
Comments