Improving Transferability of Generated Universal Adversarial Perturbations for Image Classification and Segmentation

Figure 1 gives an overview of the major components comprised within an automated vehicle: Environment perception, motion planning, and motion control [GR20]. The environment perception module collects data using several sensors to obtain an overall representation of the surroundings. This information is then processed through the motion planning module to calculate a reasonable trajectory, which is executed via the motion control module at the end. In this chapter, we concentrate on the environment perception of automated vehicles.

The environment perception of automated vehicles comprises several sensors, including radio detection and ranging (RaDAR), light detection and ranging (LiDAR), and camera [RT19]. RaDAR sensors supplement camera vision in times of low visibility, e.g., night driving, and are able to improve the detection accuracy [PTWA17]. LiDAR sensors are commonly used to make high-resolution maps, and they are capable of detecting obstacles [WXZ20]. The recent rise of deep neural networks puts a high interest to camera-based environment perception solutions, e.g., object detection and classification [ZWJW20], as well as semantic segmentation [RABA18].

In this chapter, we focus on camera-based environment perception with deep neural networks for image classification as well as for semantic segmentation in the context of AVs.

It is well known that deep neural networks are vulnerable to adversarial attacks [SZS+14]. While adversarial attacks for image classification have been widely studied, the influence of these attacks in other applications, such as semantic segmentation, has rarely been investigated [AM18, BHSFs19, BZIK20, BKV+20, KBFs20, ZBIK20a, BLK+21]. In this section, we review existing attack methods in both classification and semantic segmentation perception tasks.

There are several possible ways of categorizing an adversarial attack, e.g., targeted/non-targeted attack, and white-box/black-box attack [HM19]. In a targeted attack, the adversary generates an adversarial example to change the system’s output to a maliciously chosen target label. In a non-targeted attack, on the other hand, the attacker wants to direct the result to a label that is different from the ground truth, no matter what it is. It should be noted that in semantic segmentation, targeted attacks can be divided into static and dynamic target segmentation [MCKBF17]. Static attacks are similar to the targeted attacks in image classification. Here, the aim is to enforce the model to always output the same target semantic segmentation mask. The dynamic attack follows the objective of replacing all pixels classified with a beforehand chosen target class by its spatial nearest neighbor class. Another way of attack categorization is based on the attackers’ knowledge of the parameters and the architecture of the target model. While a white-box scenario refers to the case when an attacker has full knowledge about the underlying model, a black-box scenario implies that no information about the respective model is available. We address both targeted and non-targeted attacks as well as white-box and black-box scenarios in this chapter.

In the following, we will investigate two types of image-dependent and image-agnostic adversarial perturbations in classification and semantic segmentation models. We will also review the problem of transferability in adversarial attacks.

Image-dependent adversarial perturbations: There are various iterative, non-iterative, and generative methods for crafting image-dependent adversarial examples. Goodfellow et al. [GSS15] introduced the fast gradient sign method (FGSM), one of the first adversarial attacks. FGSM aims at computing the gradients of the source model S with respect to the (image) input \(\mathbf {x}\) and a loss J to create an adversarial example. Iterative FGSM (I-FGSM) [KGB17] iteratively applies FGSM with a small step size, while momentum iterative FGSM [DLP+18] utilizes a momentum-based optimization algorithm for stronger adversarial attacks. Another iterative attack is projected gradient descent (PGD) [MMS+18], where the main difference to I-FGSM is random restarts in the optimization process. Kurakin et al. [KGB17] proposed the least-likely class method (LLCM), in which the target is set to the least-likely class predicted by the network. Xiao et al. [XZL+18] introduced the spatial transform attack (STM) for generating adversarial examples. In STM, instead of changing the pixel values in a direct manner, spatial filters are employed to substitute pixel values maliciously. Also, some works [HM19] have employed a generative adversarial network (GAN) [GPAM+14] for generating adversarial examples.

A basic analysis on the behavior of semantic segmentation DNNs against adversarial examples was performed by Arnab et al. [AMT18]. They applied three commonly known adversarial attacks for image classification tasks, i.e., FGSM [GSS15], Iterative-FGSM [KGB17], and LLCM [KGB17], to semantic segmentation models and illustrated the vulnerability of this task. There are also some works that concentrate on sophisticated adversarial attacks that lead to more reasonable outcomes in the context of semantic segmentation [XDL+18, PKGB18].

Universal adversarial perturbations: Universal adversarial perturbations (UAPs) were firstly introduced by Moosavi-Dezfooli et al. as image-agnostic perturbations [MDFFF17]. Similar to image-dependent adversarial attacks, there are some iterative, non-iterative, and generative techniques for creating UAPs. An iterative algorithm to generate UAPs for an image classifier was presented in [MDFF+18]. The authors provided an analytical analysis of the decision boundary in DNNs based on geometry and proved the existence of small universal adversarial perturbations. Some researchers focused on generative models that can be trained for generating UAPs [HD18, RMOGVB18]. Mopuri et al. presented a network for adversary generation (NAG) [RMOGVB18], which builds upon GANs. NAG utilizes fooling and diversity loss functions to model the distribution of UAPs for a DNN image classifier. Moreover, Poursaeed et al. [PKGB18] introduced the generative adversarial perturbation (GAP) algorithm for transforming noise drawn from a uniform distribution to adversarial perturbations to conduct adversarial attacks in classification and semantic segmentation tasks. Metzen et al. [MCKBF17] proposed an iterative algorithm for semantic segmentation, which led to more realistically looking false segmentation masks.

Contrary to the previous data-dependent methods, Mopuri et al. [MGR19] introduced fast feature fool (FFF), a data-independent algorithm for producing non-targeted UAPs. FFF aims at injecting maximal adversarial energy into each layer of the source model S. This is done by the following loss function:where \(\mathbf {A}_{\ell }(\mathbf {r})\) is the mean of all feature maps of the \(\ell \)-th layer (after the activation function in layer \(\ell \)), when only the UAP \(\mathbf {r}\) is fed into the model. Note that usually \(\mathbf {x}^\text {adv} = \mathbf {x} + \mathbf {r}\), with clean image \(\mathbf {x}\), is fed into the model. This algorithm starts with a random \(\mathbf {r}\) which is then iteratively optimized. For mitigating the absence of data in producing UAPs, Mopuri et al. [MUR18] proposed class impressions (CIs), a form of reconstructed images that are obtained via simple optimization from the source model. After finding multiple CIs in the input space for each target class, they trained a generator to create UAPs. Recently, Zhang et al. [ZBIK20b] proposed a targeted UAP algorithm using random source images (TUAP-RSI) from a proxy dataset instead of the original training dataset.

In this chapter, we follow Poursaeed et al. [PKGB18] by proposing an efficient generative approach that focuses on propagating adversarial energy in a source model to generate UAPs for the task of image classification and semantic segmentation.

Transferability in black-box attacks: The ability of an adversarial example to be effective against a different, potentially unknown, target model is known as transferability. Researchers have evaluated the transferability of adversarial examples on image classifiers [MGR19, MDFFF17, PXL+20, LBX+20] and semantic segmentation networks [PKGB18, AMT18].

Regarding the philosophy behind transferability, Goodfellow et al. [GSS15] demonstrated that estimating the size of adversarial subspaces is relevant to the transferability issue. Another potential perspective lies in the similarity of decision boundaries. Learning substitute models, approximating the decision boundaries of target models, is a famous approach to attack an unknown model [PMJ+16]. Wu et al. [WWX+20] considered DNNs with skip connections and found that using more gradients from the skip connections, rather than the residual modules, allows the attacker to craft more transferable adversarial examples. Wei et al. [WLCC18] proposed to manipulate feature maps, extracted by a separate feature network, to create more transferable image-dependent perturbations using a GAN. Li et al. [LBZ+20] introduced a virtual model known as a ghost network to apply feature-level perturbations to an existing model to produce a large set of diverse models. They showed that ghost networks, together with a coupled ensemble strategy, improve the transferability of existing techniques. Wu et al. [WZTE18] empirically investigated the dependence of adversarial transferability on model-specific attributes, including model capacity, architecture, and test accuracy. They demonstrated that fooling rates heavily depend on the similarity of the source model and target model architectures.

In this chapter, we increase the tranferability of generated UAPs by including a loss term inspired by Mopuri et al. [MGR19] focusing on the adversarial energy in early layers.

We first introduce notations for image classification and semantic segmentation in a natural domain without attacks and then extend this to the adversarial domain.

Natural Domain: Consider a classifier S trained on a training set \(\mathcal {X}^\text {train}\) having M different classes. This network assigns a label \(m = S(\mathbf {x}) \in \mathcal {M}=\{1,...,M\}\) to each input image \(\mathbf {x}\) from training set \(\mathcal {X}^\text {train}\) or test set \(\mathcal {X}^\text {test}\). We assume that image \(\mathbf {x} \in \mathbb {I}^{H\times W\times C}\) is a clean image, meaning that it contains no adversarial perturbations, with height H, width W, \(C=3\) color maps, and \(\mathbb {I}=[0,1]\). Each image \(\mathbf {x}\) is tagged with a ground truth label \( \overline{m}\in \mathcal {M}\).

In semantic segmentation, given an input image \(\mathbf {x}\), we assign each pixel with a class label. In this case, the semantic segmentation network S outputs a label map \(\mathbf {m} = S(\mathbf {x}) \in \mathcal {M}^{I}\) for each input image \(\mathbf {x}=(\mathbf {x}_1, ..., \mathbf {x}_i, ..., \mathbf {x}_I)\), with \(I=H\times W\). Similar to before, each pixel \(\mathbf {x}_i \in \mathbb {I}^C\) of an image \(\mathbf {x}\) is tagged with a ground truth label \(\overline{m}_i\in \mathcal {M}\), resulting in the ground truth label map \(\overline{\mathbf {m}}\) for the entire image \(\mathbf {x}\).

Adversarial domain: Let \(\mathbf {x}^\text {adv}\) be an adversarial example that belongs to the adversarial space of the network; for example, for the network S this space is defined as \(\mathcal {X}_{S}^{\text {adv}}\). In order to have a quasi-imperceptible perturbation \(\mathbf {r}\) when added to clean images to obtain adversarial examples, i.e., \(\mathbf {x}^\text {adv} = \mathbf {x} + \mathbf {r}\), it is bounded by \(\Vert \mathbf {r}\Vert _p\le \epsilon \), with \(\epsilon \) being the supremum of a respective p-norm \(\Vert \cdot \Vert _p\).

In case of classification networks, for each \(\mathbf {x}^\text {adv} \in \mathcal {X}_{S}^{\text {adv}}\), the purpose of non-targeted attacks is to obtain \(S(\mathbf {x}^\text {adv})\ne \overline{m}\). In targeted attacks, the attacker tries to enforce \(S(\mathbf {x}^\text {adv})= \mathring{m}\ne \overline{m}\), where \(\mathring{m}\) denotes the target class the attacker aims at.

In case of semantic segmentation networks, for each \(\mathbf {x}^\text {adv} \in \mathcal {X}_{S}^{\text {adv}}\), in non-targeted attacks we aim at \(S(\mathbf {x}^\text {adv})=\mathbf {m}=(m_i)\) with \(m_{i}\ne \overline{m}_{i}\) for all \(i\in \mathcal {I} =\{1, ..., I\}\). On the other hand, in static targeted attacks, the goal is \(S(\mathbf {x}^\text {adv})= \mathring{\mathbf {m}}\ne \overline{\mathbf {m}}\), where \(\mathring{\mathbf {m}}\) denotes the target mask of the attacker.

Also, let T be the target model under attack, which is a deep neural network (DNN) with given (frozen) parameters, pretrained on some datasets to perform a specific environment perception task. We define \(\mathbf {z}\) as a random variable sampled from a distribution, which is fed to a UAP generator \(\mathbf {G}\) to produce a perturbation \(\mathbf {r}=\mathbf {G}(\mathbf {z})\). Also, J stands for a loss function, which is differentiable with respect to the network parameters and the input.

In order to improve the robustness of DNNs for environment perception, knowledge of sophisticated image-agnostic adversarial attacks is needed, capable of working in both white-box and black-box fashions.

In this section, we present our proposed approach to generate UAPs. It builds upon the adversarial perturbation generator proposed by Poursaeed et al. [PKGB18]. Unlike [PKGB18], we focus on a fooling loss function for generating effective UAPs in both white-box and black-box scenarios. We employ a pretrained DNN as the source model S, which is exposed to UAPs during training the UAP generator. Our goal is to find a UAP \(\mathbf {r}\) by an appropriate loss function, which is able to not only deceive the source model S on a training or test dataset \(\mathcal {X}^\text {train}\) or \(\mathcal {X}^\text {test}\), respectively, but also to effectively deceive a target model T, for which \(T\ne S\) holds.

Figure 2 gives an overview of our UAP generator training methodology. Let \(\mathbf {G}(\mathbf {z})= \mathbf {r}\) be the UAP generator function mapping an unstructured, random multi-dimensional input \(\mathbf {z} \sim \mathcal {U}^{H\times W \times C}\) sampled from a prior uniform distribution \(\mathcal {U}=\mathbb {I}\), onto a perturbation \(\mathbf {r}\in \mathbb {I}^{H\times W \times C}\). To obtain a p-norm scaled \(\mathbf {r}\), a preliminary obtained perturbation \(\mathbf {r}'\) is bounded by multiplying the generator network raw output \(\mathbf {r}' =\mathbf {G}'(\mathbf {z})\) with \(\min (1,\frac{\epsilon }{\Vert \mathbf {G}' (\mathbf {z})\Vert _p})\). Next, the resulting adversarial perturbation \(\mathbf {r}\) is added to an image \(\mathbf {x}\in \mathcal {X}^{\text {train}}\) before being clipped to a valid range of RGB image pixel values, resulting in an adversarial example \(\mathbf {x}^{\text {adv}}\). Finally, the generated adversarial example \(\mathbf {x}^{\text {adv}}\) is fed to a pretrained source model S to compute the adversarial loss functions based on targeted or non-targeted attack types.

To increase the model transferability of the generated UAPs, we seek similarities between different pretrained DNNs to take advantage of this property. For this, we selected some state-of-the-art classifiers such as VGG-16, VGG-19 [SZ15], ResNet-18, and ResNet-152 [HZRS16], all pretrained on ImageNet [RDS+15], to investigate their extracted feature maps in different levels. Then, we first measure the similarity of the mean feature maps of a layer between all networks over the entire ImageNet [RDS+15] validation set, using the well-known and universally applicable mean squared error (MSE).¹ Figure 3 displays the resulting heat maps. In addition, Fig. 4 shows the mean of feature representations \(\mathbf {A}_{\ell }(\mathbf {x})\) for these four pretrained classifiers computed for layer \(\ell =1\) up to \(\ell =6\) (after each activation function) for a selected input image \(\mathbf {x}\). Both figures, Figs. 3 and 4, show that the respective networks share a qualitatively and quantitatively high similarity in the first layer compared to all subsequent layers. Only for close relatives, such as VGG-16 and VGG-19, this similarity is found in later layers as well. We thus hypothesize that by applying the fast feature fool loss \(J_{1}^{\text {FFF}}\) (1) only to the first layer of the source model during training, we not only inject high adversarial energy into the first layer but also increase the transferability of the generated UAPs.

In the following, we formulate our fooling loss and consider the non-targeted and targeted attack cases (see Fig. 2).

In the non-targeted case, we want to fool the source model S so that its prediction \(S(\mathbf {x}^\text {adv})\) differs from the ground truth one-hot representation \(\overline{\mathbf {y}}\). In the simplest and most sensible possible way, researchers define the negative cross-entropy as the fooling loss for non-targeted attacks, while Poursaeed et al. [PKGB18] proposed the logarithm of this loss function.

For image classification, we define the generator fooling loss for our non-targeted attacks aswhere \(J^{\text {CE}}\) denotes the cross-entropy loss, \(\overline{\mathbf {y}}=(\overline{y}_\mu )\in \lbrace 0, 1 \rbrace ^M\) is the one-hot encoding of the ground truth label \(\overline{m}\) for image \(\mathbf {x}\), and \(\mu \) being the class index, such that \(\overline{m}=\arg \underset{\mu \in \mathcal {M}}{\max }\ \overline{y}_\mu \) . Also, let \(J^{\text {FFF}}_{1}(\mathbf {x}^{\text {adv}})\) be the fast feature fool loss of layer \(\ell =1\) (see (1)), when \(\mathbf {x}^{\text {adv}}\) is fed to the network S. Further, the cross-entropy loss is defined aswhere \(\mathbf {y} = S(\mathbf {x}^{\text {adv}}) = ({y}_\mu )\in \mathbb {I}^{M}\) is the network output vector of S with the predictions for each class \(\mu \). For optimization, we utilize Adam [KB15] in standard configuration, following [PKGB18].

For semantic segmentation, in (2) \(\mathbf {y}= S(\mathbf {x}^{\text {adv}}) \in \mathbb {I}^{H\times W \times M}\) and \(\overline{\mathbf {y}}\in \lbrace 0, 1 \rbrace ^{H\times W \times M}\) holds, and the cross-entropy loss in (3) is changed towith \(y_{i,\mu }, \overline{y}_{i,\mu }\) being the prediction and ground truth for class \(\mu \) at pixel i, respectively, and \(y_{i,\overline{m}_i}\) being the prediction at pixel i for the ground truth class \(\overline{m}_i\) of pixel i.

Different from the non-targeted case, the goal of a targeted one is to let the DNN output \(S(\mathbf {x}^{\text {adv}})\) take on values \(\mathring{\mathbf {y}}\) defined by the attacker, which usually differ from the ground truth (if source target labels align with the ground truth of a particular image, the UAP will output that ground truth label). Hence, the attacker aims to decrease the cross-entropy loss with respect to a target until the source model S predicts the selected target class with high confidence. As before, we add the fast feature fool loss in the first layer to boost the transferability of the targeted generated UAP.

For image classification, our generator fooling loss for targeted attacks iswhere \(\mathring{\mathbf {y}}\in \lbrace 0,1\rbrace ^{M}\) is the one-hot encoding of the target label \(\mathring{m}\ne \overline{m}\). Note that the sign of the cross entropy is flipped compared to the non-targeted case (2). Then, similar to the non-targeted attack, the Adam optimizer is utilized.

If we consider semantic segmentation, the ground truth \(\mathring{\mathbf {y}}\) in (5) becomes a one-hot encoded semantic segmentation mask \(\mathring{\mathbf {y}}\in \lbrace 0, 1 \rbrace ^{H\times W \times M}\).

Dataset: We use the ImageNet dataset [RDS+15], which is a large collection of human-annotated images. For all our experiments, a universal adversarial perturbation is computed for a set of 10,000 images taken from the ImageNet training set \(\mathcal {X}^{\text {train}}\) (i.e., 10 images per class) and the results are reported on the ImageNet validation set \(\mathcal {X}^{\text {val}}\) (50,000 images).

Network architectures: There are several design options regarding the architecture choices for generator \(\mathbf {G}\) and source model S. For our generator, we follow [ZPIE17] and [PKGB18] and choose the ResNet generator from [JAFF16], which consists of some convolution layers for downsampling, followed by some residual blocks, before performing upsampling using transposed convolutions. As topology for the source model S, we utilize the same set of pretrained image classifiers as for the target model T, i.e., VGG-16, VGG-19 [SZ15], ResNet-18, ResNet-152 [HZRS16], and also GoogleNet [SLJ+15].

Evaluation metrics: We use the fooling rate as our metric to assess the performance of our crafted UAPs on DNN image classifiers [MDFFF17, PKGB18, MUR18, MGR19]. In the case of non-targeted attacks, it is the percentage of input images for which \(T(\mathbf {x}^{\text {adv}})\ne T(\mathbf {x})\) holds. For targeted attacks, we calculate the top-1 target accuracy, which can be understood as the percentage of adversarial examples, that is classified “correctly" as the target class as desired by the attacker.

According to Fig. 2, we train our model with the non-targeted fooling loss (2). For tuning the hyperparameter \(\alpha \), the weight of our novel adversarial loss components, we utilized a second training set of 10,000 random images (again 10 images per class) taken from the ImageNet training set which is disjoint from both the training and the validation dataset. Table 1 shows that the best \(\alpha \) for non-targeted attacks, on average over all model topologies, is \(\alpha =0.7\).

For white-box attacks, where \(S=T\) holds, results on the ImageNet validation set for two different norms are given in Table 2. The maximum permissible \(L_p\) norm of the perturbations for \(p=2\) and \(p=\infty \) is set to be \(\epsilon = 2000\) and \(\epsilon = 10\), respectively, following [MDFFF17]. As Moosavi-Dezfooli et al. [MDFFF17] pointed out, these values are selected to acquire a perturbation whose norm is remarkably smaller than the average image norms in the ImageNet dataset to obtain quasi-imperceptible adversarial examples. The results in Table 2 show that the proposed method is successful in the white-box setting. For the \(L_{\infty }\) norm, all reported fooling rate numbers are above 90%. To illustrate that our adversarial examples are quasi-imperceptible to humans, we illustrate some adversarial examples with their respective scaled UAPs in Fig. 5.

In Table 3, we compare our proposed approach in generating non-targeted UAPs with state-of-the-art methods, i.e., fast feature fool (FFF, as originally proposed, on all layers) [MGR19], class impressions (CIs) [MUR18], universal adversarial perturbation (UAP) [MDFFF17], generative adversarial perturbation (GAP) [PKGB18], network for adversary generation (NAG) [RMOGVB18], and targeted UAP-random source image (TUAP-RSI) [ZBIK20b]. In these experiments, we again consider the white-box setting, i.e., \(S=T\). Our proposed approach achieves a new state-of-the-art performance on almost all models on both \(L_p\) norms, being on average 4% absolute better in fooling rate with the \(L_2\) norm, and at least 0.26% absolute better with the \(L_{\infty }\) norm.

In Table 4 we investigate the white-box (\(S=T\)) and black-box (\(S \ne T\)) capability of our proposed method through various combinations of source and target models. Note that the rightmost column represents an average over the fooling rates in the black-box settings and thus indicates the transferability of our proposed method. Overall, in the white-box and black-box settings, we achieve fooling rates of over 90% and 55%, respectively. We also compare the transferability of our produced UAPs with the same state-of-the-art methods as before. The results for these experiments are shown in Table 5, where VGG-16, ResNet-152, and GoogleNet are used as the source model in Table 5a–c, respectively. It turns out to be advisable to choose a deep network as the source model (ResNet-152); since then our performance on the unseen VGG-16 and VGG-19 target models is about 12% absolute better than earlier state of the art (\(L_{\infty }\) norm).

For investigating the generalization power of UAPs w.r.t. unseen data, we evaluate the influence of the size of the training dataset \(\mathcal {X}^\text {train}\) on the quality of UAPs in conducting white-box and black-box attacks. Figure 6 shows the fooling rates obtained for VGG-16 as the target model T, on the ImageNet validation set for different sizes of \(\mathcal {X}^\text {train}\). The results show that by using a dataset \(\mathcal {X}^\text {train}\) containing only 1000 images, our approach leads to a fooling rate of more than 60% on the ImageNet validation dataset in both the white-box and black-box settings. Additionally, the number of images in \(\mathcal {X}^\text {train}\) turns out to be more vital for the fooling rate of black-box attacks as compared to white-box attacks.

To examine the impact of the layer which is used in our loss function, we utilized different layers in (1), then applied them in the loss function (2) to train the source model for generating UAPs. In practice, we are interested in the impact the layer position has. Figure 7 shows the fooling rate in white-box and black-box settings for \(L_{\infty }\) and \(L_2\), when different layers from \(\ell =1\) to \(\ell =6\) are applied in (1) and (2). This figure shows that choosing deeper layers leads to a decreasing trend in the fooling rate. Also, this trend is stronger in black-box attacks, where a more than 20% drop in the attack success rate can be observed. This indicates that it is advisable to choose earlier layers, in particular the first layer, in generating UAPs to obtain both an optimal fooling rate and transferability.

In this section, we apply the targeted fooling loss (5), again with \(\alpha =0.7\), for training the generator in Fig. 2. We assume the chosen \(\alpha \) is appropriate for targeted attacks as well and thus dispense further investigations. If results are good, then this supports some robustness w.r.t. the choice of \(\alpha \). Figure 8 depicts two examples of our targeted UAPs, some original images and respective adversarial examples. In these experiments, the fooling rate (top-1 target accuracy) on the validation set for the target class \(\mathring{m}=919\) (street sign) and \(\mathring{m}=920\) (traffic light, traffic signal, stoplight) are 63.2 and 57.83%, respectively, which underlines the effectiveness of our approach.

For assessing the generalization power of our proposed method across different target classes and comparison with GAP [PKGB18], we followed Poursaeed et al. and used 10 randomly sampled classes. The resulting average top-1 target accuracy, when the adversarial perturbation is bounded by \(L_{\infty }(\mathbf {r})\le \epsilon = 10\), is 66.57%, which is significantly higher than the one reported for GAP [PKGB18] with 52.0%.

To demonstrate the generalization power of our targeted UAPs w.r.t. unseen data, we visualize the attack success rate obtained by the source model VGG-16, in white-box and black-box settings, on the Image-Net validation dataset for different sizes of \(\mathcal {X}^\text {train}\) in Fig. 9. For instance, with \(\mathcal {X}^\text {train}\) containing 10,000 images, we are able to fool the target model on over 20% of the images in the ImageNet validation set. It should be noted that training the generator \(\mathbf {G}\) to produce a single UAP forcing the target model to output a specific target class \(\mathring{m}\) is an extremely challenging task. However, we particularly observe that utilizing 10,000 training images again seems to be sufficient for a white-box attack.

Dataset: We conduct our experiments on the widely known Cityscapes dataset [COR+16]. It contains pixel-level annotations of 5,000 high-resolution images (2,975 training, 500 validation, and 1,525 test images) being captured in urban street scenes. The original images have a resolution of \(2048 \times 1024\) pixels, that we downsample to \(1024 \times 512\) for our experiments [MCKBF17, PKGB18].

Network architecture: Regarding the architecture of the source model S, we use FCN-8 [LSD15] for white-box attacks, and ERFNet [RABA18] for the black-box setting. FCN-8 consists of an encoder part which transforms an input image into a low-resolution semantic representation and a decoder part which recovers the high spatial resolution of the image by fusing different levels of feature representations together. ERFNet also consists of an encoder-decoder structure, but without any bypass connections between the encoder and the decoder. Additionally, residual units are used with factorized convolutions to obtain a more efficient computation.

In our experiments, we consider FCN-8 [LSD15] as our segmentation target model T, and use \(L_{\infty }\) norm, to be comparable with [MCKBF17, PKGB18].

Evaluation metrics: To assess the performance of a semantic segmentation network, we used the mean intersection-over-union (mIoU) [COR+16]. It is defined aswith class \(\mu \in \mathcal {M}\), class-specific true positives \(\text {TP}_{\mu }\), false positives \(\text {FP}_{\mu }\), and false negatives \(\text {FN}_{\mu }\). For assessing the impact of non-targeted adversarial attacks on semantic segmentation, we compute the mIoU on adversarial examples [AMT18].

To measure the attack success of our targeted attack, we compute the pixel accuracy (PA) between the prediction \(\mathbf {m} = T(\mathbf {x}^\text {adv})\) and the target \(\mathring{\mathbf {m}}\) [PKGB18]. In this chapter, we restrict our analysis to the static target segmentation scenario [MCKBF17] and use the same target mask as in [PKGB18, MCKBF17] (see also Fig. 11).

For the non-targeted case, we train our model with the non-targeted adversarial loss function (2), where we use (4) as the cross-entropy loss. The maximum permissible \(L_p\) norm of the perturbations for \(p=\infty \) is set to \(\epsilon \in \lbrace 2, 5, 10, 20 \rbrace \). We report results for both our method and GAP [PKGB18] on the Cityscapes validation set in Table 6a. It can be observed that across different values for \(\epsilon \) our method is superior to GAP in terms of decreasing the mIoU of the underlying target model.

We visualize the effect of the generated non-targeted UAPs in Fig. 10 by illustrating the original image, the UAP, the resulting adversarial example, the prediction on the original image, the ground truth mask, and the resulting segmentation output. Here, the maximum permissible \(L_p\) norm of the perturbations \(p=\infty \) is set to \(\epsilon = 5\).

For investigating the transferability of our generated UAPs, we use a UAP optimized on ERFNet as the source model S and test it on the target model T being FCN-8. Table 6b reports black-box attack results for our method compared to the attack method GAP [PKGB18]. Our non-targeted UAPs decrease the mIoU of the FCN-8 on the Cityscapes dataset more than GAP [PKGB18] does, in all different ranges of adversarial perturbations (\(\epsilon \in \{2,5,10,20\}\)). These results illustrate the effectivity of the generated perturbation.

In the targeted case, we aim at finding a UAP which forces the segmentation source model S and the segmentation target model T to predict a specific target mask \(\mathring{\mathbf {m}}\). We now train our model with the targeted adversarial loss function (5), using \(J^{\text {CE}}(\mathbf {y}, \mathring{\mathbf {y}})\) according to (4) for the cross-entropy loss. We apply the same target \(\mathring{\mathbf {y}}\) as in [PKGB18], see \(\mathring{\mathbf {m}}\) in Fig. 11e.

Figure 11 depicts an original image, our generated targeted UAP, the respective adversarial example, the prediction on the original image, the target segmentation mask, and the prediction on the adversarial example. The results show that the generated UAP resembles the target mask and is able to fool the FCN-8 in a way that it now outputs the target segmentation mask.

We compare our targeted attack with two state-of-the-art methods in Table 7. While our method performs comparably well as state of the art for weak attacks, in medium to strong attacks we outperform both GAP [PKGB18] and UAP-Seg [MCKBF17].