Skip to main content
Top
Published in: International Journal of Machine Learning and Cybernetics 11/2021

04-01-2021 | Original Article

Adversarial examples: attacks and defenses in the physical world

Authors: Huali Ren, Teng Huang, Hongyang Yan

Published in: International Journal of Machine Learning and Cybernetics | Issue 11/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Deep learning technology has become an important branch of artificial intelligence. However, researchers found that deep neural networks, as the core algorithm of deep learning technology, are vulnerable to adversarial examples. The adversarial examples are some special input examples which were added small magnitude and carefully crafted perturbations to yield erroneous results with extremely confidence. Hence, they bring serious security risks to deep-learning-based systems. Furthermore, adversarial examples exist not only in the digital world, but also in the physical world. This paper presents a comprehensive overview of adversarial attacks and defenses in the real physical world. First, we reviewed these works that can successfully generate adversarial examples in the digital world, analyzed the challenges faced by applications in real environments. Then, we compare and summarize the work of adversarial examples on image classification tasks, target detection tasks, and speech recognition tasks. In addition, the relevant feasible defense strategies are summarized. Finally, relying on the reviewed work, we propose potential research directions for the attack and defense of adversarial examples in the physical world.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Show more products
Literature
1.
go back to reference LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRef LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRef
2.
go back to reference Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Advances in neural information processing systems. Springer, New York, pp 3104–3112 Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Advances in neural information processing systems. Springer, New York, pp 3104–3112
3.
go back to reference He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
4.
go back to reference Senior A, Vanhoucke V, Guyen P, Sainath T et al (2012) Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process Mag 29:82 Senior A, Vanhoucke V, Guyen P, Sainath T et al (2012) Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process Mag 29:82
5.
go back to reference Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. Springer, New York, pp 1097–1105 Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. Springer, New York, pp 1097–1105
6.
go back to reference Athalye A, Engstrom L, Ilyas A, Kwok K (2018) Synthesizing robust adversarial examples. In: International conference on machine learning. PMLR, pp 284– 293 Athalye A, Engstrom L, Ilyas A, Kwok K (2018) Synthesizing robust adversarial examples. In: International conference on machine learning. PMLR, pp 284– 293
7.
go back to reference Sun L, Tan M, Zhou Z (2018) A survey of practical adversarial example attacks. Cybersecurity 1(1):9CrossRef Sun L, Tan M, Zhou Z (2018) A survey of practical adversarial example attacks. Cybersecurity 1(1):9CrossRef
8.
go back to reference Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013) Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013) Intriguing properties of neural networks. arXiv preprint arXiv:​1312.​6199
9.
go back to reference Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. Comput Sci Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. Comput Sci
10.
go back to reference Kurakin A, Goodfellow I, Bengio S (2016) Adversarial examples in the physical world Kurakin A, Goodfellow I, Bengio S (2016) Adversarial examples in the physical world
11.
go back to reference Moosavi-Dezfooli SM, Fawzi A, Fawzi O, Frossard P (2017) Universal adversarial perturbations Moosavi-Dezfooli SM, Fawzi A, Fawzi O, Frossard P (2017) Universal adversarial perturbations
12.
go back to reference Moosavi-Dezfooli S-M, Fawzi A, Frossard P (2016) Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2574–2582 Moosavi-Dezfooli S-M, Fawzi A, Frossard P (2016) Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2574–2582
13.
go back to reference Carlini N, Wagner D (2017) Towards evaluating the robustness of neural networks. In: 2017 IEEE symposium on security and privacy (sp). IEEE, pp 39–57 Carlini N, Wagner D (2017) Towards evaluating the robustness of neural networks. In: 2017 IEEE symposium on security and privacy (sp). IEEE, pp 39–57
14.
go back to reference Xie C, Zhang Z, Zhou Y, Bai S, Wang J, Ren Z, Yuille AL (2019) Improving transferability of adversarial examples with input diversity. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2730–2739 Xie C, Zhang Z, Zhou Y, Bai S, Wang J, Ren Z, Yuille AL (2019) Improving transferability of adversarial examples with input diversity. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2730–2739
15.
go back to reference Dong Y, Liao F, Pang T, Su H, Zhu J, Hu X, Li J (2018) Boosting adversarial attacks with momentum. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9185–9193 Dong Y, Liao F, Pang T, Su H, Zhu J, Hu X, Li J (2018) Boosting adversarial attacks with momentum. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9185–9193
16.
go back to reference Miyato T, Dai AM, Goodfellow I (2016) Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725 Miyato T, Dai AM, Goodfellow I (2016) Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:​1605.​07725
17.
go back to reference Zheng S, Song Y, Leung T, Goodfellow I (2016) Improving the robustness of deep neural networks via stability training. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4480–4488 Zheng S, Song Y, Leung T, Goodfellow I (2016) Improving the robustness of deep neural networks via stability training. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4480–4488
18.
go back to reference Buckman J, Roy A, Raffel C, Goodfellow I (2018) Thermometer encoding: one hot way to resist adversarial examples. In: International conference on learning representations Buckman J, Roy A, Raffel C, Goodfellow I (2018) Thermometer encoding: one hot way to resist adversarial examples. In: International conference on learning representations
19.
go back to reference Song Y, Kim T, Nowozin S, Ermon S, Kushman N (2017) Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:1710.10766 Song Y, Kim T, Nowozin S, Ermon S, Kushman N (2017) Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:​1710.​10766
20.
21.
22.
go back to reference Akhtar N, Liu J, Mian A (2018) Defense against universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3389–3398 Akhtar N, Liu J, Mian A (2018) Defense against universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3389–3398
23.
go back to reference Zhou H, Li W, Zhu Y, Zhang Y, Yu B, Zhang L, Liu C (2018) Deepbillboard: systematic physical-world testing of autonomous driving systems Zhou H, Li W, Zhu Y, Zhang Y, Yu B, Zhang L, Liu C (2018) Deepbillboard: systematic physical-world testing of autonomous driving systems
24.
go back to reference Patel N, Krishnamurthy P, Garg S, Khorrami F (2019) Adaptive adversarial videos on roadside billboards: dynamically modifying trajectories of autonomous vehicles. In: 2019 IEEE/RSJ International conference on intelligent robots and systems (IROS) Patel N, Krishnamurthy P, Garg S, Khorrami F (2019) Adaptive adversarial videos on roadside billboards: dynamically modifying trajectories of autonomous vehicles. In: 2019 IEEE/RSJ International conference on intelligent robots and systems (IROS)
25.
go back to reference Sato T, Shen J, Wang N, Jia YJ, Lin X, Chen QA (2020) Security of deep learning based lane keeping system under physical-world adversarial attack Sato T, Shen J, Wang N, Jia YJ, Lin X, Chen QA (2020) Security of deep learning based lane keeping system under physical-world adversarial attack
26.
go back to reference Thys S, Van Ranst W, Goedemé T (2019) Fooling automated surveillance cameras: adversarial patches to attack person detection Thys S, Van Ranst W, Goedemé T (2019) Fooling automated surveillance cameras: adversarial patches to attack person detection
27.
go back to reference Liu A, Liu X, Fan J, Ma Y, Tao D (2019) Perceptual-sensitive gan for generating adversarial patches. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 1028–1035 Liu A, Liu X, Fan J, Ma Y, Tao D (2019) Perceptual-sensitive gan for generating adversarial patches. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 1028–1035
28.
go back to reference Li J, Schmidt FR, Kolter JZ (2019) Adversarial camera stickers: a physical camera-based attack on deep learning systems Li J, Schmidt FR, Kolter JZ (2019) Adversarial camera stickers: a physical camera-based attack on deep learning systems
29.
go back to reference Zhou Z, Tang D, Wang X, Han W, Liu X, Zhang K (2018) Invisible mask: practical attacks on face recognition with infrared Zhou Z, Tang D, Wang X, Han W, Liu X, Zhang K (2018) Invisible mask: practical attacks on face recognition with infrared
30.
go back to reference Cao Y, Xiao C, Cyr B, Zhou Y, Park W, Rampazzi S, Chen QA, Fu K, Mao ZM (2019) Adversarial sensor attack on lidar-based perception in autonomous driving. In: Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp 2267–2281 Cao Y, Xiao C, Cyr B, Zhou Y, Park W, Rampazzi S, Chen QA, Fu K, Mao ZM (2019) Adversarial sensor attack on lidar-based perception in autonomous driving. In: Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp 2267–2281
31.
go back to reference Eykholt K, Evtimov I, Fernandes E, Li B, Rahmati A, Xiao C, Prakash A, Kohno T, Song D (2018) Robust physical-world attacks on deep learning models. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR) Eykholt K, Evtimov I, Fernandes E, Li B, Rahmati A, Xiao C, Prakash A, Kohno T, Song D (2018) Robust physical-world attacks on deep learning models. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR)
32.
go back to reference Sharif M, Bhagavatula S, Bauer L, Reiter MK (2016) Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp 1528–1540 Sharif M, Bhagavatula S, Bauer L, Reiter MK (2016) Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp 1528–1540
33.
go back to reference Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2017) Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp 506–519 Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2017) Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp 506–519
34.
go back to reference Papernot N, McDaniel P, Jha S, Fredrikson M, Celik ZB, Swami A (2016) The limitations of deep learning in adversarial settings. In: 2016 IEEE European symposium on security and privacy (EuroS & P). IEEE, pp 372–387 Papernot N, McDaniel P, Jha S, Fredrikson M, Celik ZB, Swami A (2016) The limitations of deep learning in adversarial settings. In: 2016 IEEE European symposium on security and privacy (EuroS & P). IEEE, pp 372–387
35.
go back to reference Su J, Vargas DV, Sakurai K (2019) One pixel attack for fooling deep neural networks. IEEE Trans Evol Comput 23(5):828–841CrossRef Su J, Vargas DV, Sakurai K (2019) One pixel attack for fooling deep neural networks. IEEE Trans Evol Comput 23(5):828–841CrossRef
36.
go back to reference LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRef LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRef
37.
38.
go back to reference Girshick R (2015) Fast R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448 Girshick R (2015) Fast R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448
39.
go back to reference Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp 91–99 Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp 91–99
40.
go back to reference He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969 He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969
41.
go back to reference Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271 Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271
42.
go back to reference Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Ssd: Single shot multibox detector, in European conference on computer vision. Springer, pp. 21–37 Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Ssd: Single shot multibox detector, in European conference on computer vision. Springer, pp. 21–37
43.
go back to reference Wu Z, Lim SN, Davis L, Goldstein T (2019) Making an invisibility cloak: real world adversarial attacks on object detectors Wu Z, Lim SN, Davis L, Goldstein T (2019) Making an invisibility cloak: real world adversarial attacks on object detectors
44.
go back to reference Cao Y, Xiao C, Yang D, Fang J, Yang R, Liu M, Li B (2019) Adversarial objects against lidar-based autonomous driving systems Cao Y, Xiao C, Yang D, Fang J, Yang R, Liu M, Li B (2019) Adversarial objects against lidar-based autonomous driving systems
45.
go back to reference Tu J, Ren M, Manivasagam S, Liang M, Yang B, Du R, Cheng F, Urtasun R (2020) Physically realizable adversarial examples for lidar object detection Tu J, Ren M, Manivasagam S, Liang M, Yang B, Du R, Cheng F, Urtasun R (2020) Physically realizable adversarial examples for lidar object detection
46.
go back to reference Carlini N, Wagner D (2018) Audio adversarial examples: targeted attacks on speech-to-text. In: 2018 IEEE security and privacy workshops (SPW). IEEE, pp 1–7 Carlini N, Wagner D (2018) Audio adversarial examples: targeted attacks on speech-to-text. In: 2018 IEEE security and privacy workshops (SPW). IEEE, pp 1–7
47.
go back to reference Hannun A, Case C, Casper J, Catanzaro B, Diamos G, Elsen E, Prenger R, Satheesh S, Sengupta S, Coates et al A (2014) Deep speech: scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 Hannun A, Case C, Casper J, Catanzaro B, Diamos G, Elsen E, Prenger R, Satheesh S, Sengupta S, Coates et al A (2014) Deep speech: scaling up end-to-end speech recognition. arXiv preprint arXiv:​1412.​5567
48.
go back to reference Zhang G, Yan C, Ji X, Zhang T, Zhang T, Xu W (2017) Dolphinattack: Inaudible voice commands. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 103–117 Zhang G, Yan C, Ji X, Zhang T, Zhang T, Xu W (2017) Dolphinattack: Inaudible voice commands. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 103–117
49.
go back to reference Taori R, Kamsetty A, Chu B, Vemuri N (2019) Targeted adversarial examples for black box audio systems. In: 2019 IEEE security and privacy workshops (SPW). IEEE, pp 15–20 Taori R, Kamsetty A, Chu B, Vemuri N (2019) Targeted adversarial examples for black box audio systems. In: 2019 IEEE security and privacy workshops (SPW). IEEE, pp 15–20
50.
go back to reference Wu T, Tong L, Vorobeychik Y (2019) Defending against physically realizable attacks on image classification Wu T, Tong L, Vorobeychik Y (2019) Defending against physically realizable attacks on image classification
51.
go back to reference Xu Z, Yu F, Chen X (2020) Lance: A comprehensive and lightweight CNN defense methodology against physical adversarial attacks on embedded multimedia applications. In: 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, pp 470–475 Xu Z, Yu F, Chen X (2020) Lance: A comprehensive and lightweight CNN defense methodology against physical adversarial attacks on embedded multimedia applications. In: 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, pp 470–475
Metadata
Title
Adversarial examples: attacks and defenses in the physical world
Authors
Huali Ren
Teng Huang
Hongyang Yan
Publication date
04-01-2021
Publisher
Springer Berlin Heidelberg
Published in
International Journal of Machine Learning and Cybernetics / Issue 11/2021
Print ISSN: 1868-8071
Electronic ISSN: 1868-808X
DOI
https://doi.org/10.1007/s13042-020-01242-z

Other articles of this Issue 11/2021

International Journal of Machine Learning and Cybernetics 11/2021 Go to the issue