Ecological Analogy for Generative Adversarial Networks and Diversity Control

In GANs, we have two learning dynamics. One is for the generator and the other is for the discriminator. The generator is a transformation from an input, z , to the output, x , which has the same form of given dataset. The discriminator evaluates the input, x .

$$\begin{align} \boldsymbol{x}&=G(\boldsymbol{z},\boldsymbol{\theta}_g) \end{align} \tag{ 1 }$$

$$\begin{align} y&=D(\boldsymbol{x},\boldsymbol{\theta}_d), \end{align} \tag{ 2 }$$

where $\boldsymbol{\theta}_g$ and $\boldsymbol{\theta}_d$ are parameters of the networks. In the training of GAN, we usually input random numbers as z into the generator. As the input for the discriminator, we use samples from the given dataset, ${\boldsymbol{x}}_i\in{\cal X}$, and outputs from the generator, $\boldsymbol{x} = G(\boldsymbol{z})$. We have two loss functions to be optimized, therefore. We should minimize a loss function, L_g , for the generator,

$$\begin{equation} \dot{\boldsymbol{\theta}_g}\propto-\nabla_{\theta_g}L_g(D(G(\boldsymbol{z}_i,\boldsymbol{\theta}_g),\boldsymbol{\theta}_d),1). \end{equation} \tag{ 3 }$$

The input, z _i , is sampled random number. The generated sample, $G(\boldsymbol{z}_i)$, should be discriminated as genuine, $D(G(\boldsymbol{z}_i)) = 1$, as the result of the training. We also have one more loss function, L_d , for the discriminator. We can write an evolution function for the discriminator with the loss functions,

$$\begin{align} \nonumber\dot{\boldsymbol{\theta}_d}\propto & -\nabla_{\theta_d}L_d(D(\boldsymbol{X}_i,\boldsymbol{\theta}_d),1)[2pt]\nonumber\\ & -\nabla_{\theta_d}L_g(D(G(\boldsymbol{z}_i,\boldsymbol{\theta}_g),\boldsymbol{\theta}_d),0)). \end{align} \tag{ 4 }$$

In this equation, we want to optimize the discriminator to evaluate the training data, $\boldsymbol{X}_i\in{\cal X}$, as genuine, $D(\boldsymbol{X}_i) = 1$, and the generated one as fake, $D(G(\boldsymbol{z}_i)) = 0$. In a training step, called as 'epoch', we usually update the parameters, $\boldsymbol{\theta}_g$ and $\boldsymbol{\theta}_d$, with the randomly shuffled sub-sets of the training data. In this way, the dynamics is usually somehow stochastic along the adopted optimizer [9–15].

Here we consider a very simple ecological system, where two species of animals, A and B, are there. The animals want to migrate to the place with many foods. The animals A move their distribution along the environmental food condition. In other words, we can regard it as a herbivore. When they are eaten by other animals, they want to avoid them. We can express the distribution of the foods with a function of the place, $R(\boldsymbol{x})$. We assume this function is fixed. As an assumption on the distribution of the animal A, we simply describe the distribution in a equation,

$$\begin{equation} P_A(\boldsymbol{x})\propto\sum_i K(\boldsymbol{x},\boldsymbol{a}_i,h), \end{equation} \tag{ 5 }$$

where K is a distribution kernel around the position, a _i . In this paper, we can assume the kernel as the following,

$$\begin{equation} K(\boldsymbol{x},\boldsymbol{y},h)\sim\exp\left(-\frac{(\boldsymbol{x}-\boldsymbol{y})^2}{2h}\right). \end{equation} \tag{ 6 }$$

One more assumption for animals B, we use a simple distribution again,

$$\begin{equation} P_B(\boldsymbol{x})\propto\sum_i K(\boldsymbol{x},\boldsymbol{b}_i,h). \end{equation} \tag{ 7 }$$

They live around some spots, b _i . We assume the animal B eat the other, the animal A, and they are carnivores therefore. As the migration dynamics, we can assume a simple dynamical system on their habitats, a_i and b_i ,

$$\begin{align} \dot{\boldsymbol{a}_i}\propto&\nabla R(\boldsymbol{x})-\nabla \sum_i K(\boldsymbol{x},\boldsymbol{b}_i,h) \end{align} \tag{ 8 }$$

$$\begin{align} \dot{\boldsymbol{b}_j}\propto&\nabla \sum_i K(\boldsymbol{x},\boldsymbol{a}_i,h). \end{align} \tag{ 9 }$$

In the similar way, we assume the environmental food distribution can be written in the following equation,

$$\begin{equation} R(\boldsymbol{x})\propto\sum_i K(\boldsymbol{x},\boldsymbol{c}_i,h). \end{equation} \tag{ 10 }$$

We can easily understand this ecological system with the ecological niches, a _i , b _i and c _i .

In the next section, we study the ecological analogy of GAN based on the two dynamical systems, introduced here. We also demonstrate the GAN has such an analogical dynamics with the known data set MNIST [8]. Furthermore, we introduce a control method for GAN and demonstrate it as well.

We introduce some assumptions for the dynamics of GAN and show an ecological interpretation for that, here. We usually assume a deep network for the generator, $\boldsymbol{x} = G(\boldsymbol{z})$, by which we can get a Monte-Carlo sample, x , with a random input, z . In this meaning, this realizes a distribution, $P_G(\boldsymbol{x})$, of samples to be generated. As a very easy case, we introduce an assumption for that,

$$\begin{equation} P_G(\boldsymbol{x})\propto\sum_iK(\boldsymbol{x},\boldsymbol{g}_i,h). \end{equation} \tag{ 11 }$$

In other words, we assume it has more samples around the points, g _i , and those are parameters to be optimized. Similarly, we introduce one more assumption on the discriminator, $D(\boldsymbol{x})$. Since the discriminator determines the map, by which the points are discriminated as genuine, D = 1, or not, D = 0, we can model it as an easy mixed distribution,

$$\begin{equation} D(\boldsymbol{x})\propto\sum_iK(\boldsymbol{x},\boldsymbol{d}_i,h). \end{equation} \tag{ 12 }$$

This discriminator evaluates a point, x , as genuine, if it is near the one of the centers, $\{\boldsymbol{d}_i\}$. On the other hand, the point far away from all of them, $\{\boldsymbol{d}_i\}$, should be discriminated as fake. The centers, d _i , are the parameters to be optimized to minimize the loss, L_d .

We rewrite the evolution equations for GAN with these assumptions. We can sample a generated one, $\boldsymbol{x} = \boldsymbol{g}_i+\boldsymbol{\epsilon}$. With the sample, we can evaluate the loss, L_g ,

$$\begin{equation} L_g\propto-\sum_jK(\boldsymbol{g}_i+\boldsymbol{\epsilon},\boldsymbol{d}_j,h)\sim -K(\boldsymbol{g}_i,\boldsymbol{d}_j^*,h). \end{equation} \tag{ 13 }$$

The parameter, $\boldsymbol{d}_j^*$, is the nearest neighbor one from the sample, $\boldsymbol{g}_i+\boldsymbol{\epsilon}$. The learning dynamics can be written with this,

$$\begin{equation} \dot{\boldsymbol{g}_i}\propto\nabla_{g_i}K(\boldsymbol{g}_i,\boldsymbol{d}_j^*,h). \end{equation} \tag{ 14 }$$

In addition, we assume the training data distribution, ${\cal X}$, can be written with some representative points,

$$\begin{equation} P(\boldsymbol{X})\propto\sum_iK(\boldsymbol{X},\boldsymbol{T}_i,h). \end{equation} \tag{ 15 }$$

With this expression, we can write the evolution of the discriminator, $D(\boldsymbol{x})$, in the following,

$$\begin{align} \dot{\boldsymbol{d}_j}\propto&\nabla_{d_j}\sum_{i}K(\boldsymbol{T}_i,\boldsymbol{d}_j,h) \end{align} \tag{ 16 }$$

$$\begin{align} &-\nabla_{d_j}\sum_{i}K(\boldsymbol{g}_i,\boldsymbol{d}_j,h) \end{align} \tag{ 17 }$$

$$\begin{align} \sim&\nabla_{d_j}K(\boldsymbol{T}_i^*,\boldsymbol{d}_j,h)-\nabla_{d_j}K(\boldsymbol{g}_i^*,\boldsymbol{d}_j,h), \end{align} \tag{ 18 }$$

where the points, $\boldsymbol{T}_i^*$ and $\boldsymbol{g}_i^*$, are the nearest neighbor one from the parameter, d _j . We now get the simple model for learning dynamics of GAN. The dynamics shows striking similarity with the ecological dynamics, which we defined in the previous section. In other words, we can interpret the dynamics with prey-predator one.

As we notice, the ecological dynamics does never assure the diversity of the distribution of animals. The animals, both A and B, can stay around only one niche, c _i . In the similar way, we can expect this type of solutions can be realized in GAN, as well. In other words, the ideal solution of GAN can be dynamically unstable or not the unique solution.

Here, we show results with MNIST. For simplicity, we firstly show the results with reduced MNIST. In the MNIST, we have labeled data set, consists of pairs of an image and the digit. Each image includes a manually written digit. We randomly select pairs or triplets of digits as the reduced cases. In the case of paired MNIST, we randomly select two digits and all images labeled as them are used. In the case of triplet one, three digits are randomly selected and the training data consists of all of the images with the digits, similarly. The networks, the generator and the discriminator, learn how to generate new digit images and discriminate whether an input image is from the training data or generated ones, respectively.

In figures 1–3, we show the results of pair cases with 100 iterated tests. We evaluated the distance between generated samples, x , and MNIST images, X _i , to determine the class label of the sample. We selected the nearest neighbor MNIST image, in the training data set, for each generated one and classified it as the same class of that from MNIST. In this way, all generated ones can be classified into one of digits. The results show time series of of the class ratios, r₀ and r₁. We trained the network for 500 epochs in all cases. We used Adam optimizer with the learning rate, 0.001[15]. We calculated this with the simplest GAN with a full connected hidden layer in generator and discriminator. The dynamics is oscillatory and those are around the mid-ratio, $0.4 \sim 0.5$, and in the biased region, ${\sim}0.1$, in the left of the figure 1. We only show one of them, r₀ and $r_1 = 1-r_0$, which is less at the end of training. In the final population, right in it, we have four peaks in the end ratio. Among them, we can confirm two peaks, in the mid-range, with weak bias. The generated distributions, with such a weakly biased ratio, can cover both of selected digits. On the contrary, almost all of the generated images are classified into either of the digits in the cases with the highly biased ratio.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We also checked the repeatability, in figures 2 and 3. From the 100 iterated tests, we can select some ones with the same pair. Among them, we show the results of pairs, $(0,2)$ and $(4,8)$. In the first ones, we can always confirm '0' is stronger than '2'. On the other hand, in the cases with the pair, $(4,8)$, we can confirm the non-biased ratios in the both cases. These results suggest the attractors are specific to the pair.

In figure 4, we show the results of triplet cases. We calculated and evaluated this in the same way of the pair cases, for 30 iterated tests. Here, we also plotted the ratio between triples. We randomly selected three digits and all of the MNIST images with the selected ones are used for training. The dots of initial and end ratio are shown. We can confirm the initial dots are centered in the ternary plot. On the contrary, the end dots tend to distribute widely in the area. Especially, we observe more dots around edges.

Zoom In Zoom Out Reset image size

The samples of learning dynamics are shown in the figure 5. We can confirm a convergent dynamics to non-biased distributions, in the left. In the right, the biased distributions are realized as the result of training. We can find both non-biased and biased results, but biased ones are more, as confirmed in the figure 4.

Zoom In Zoom Out Reset image size

In figure 6, we show sampled generated images of the case with full MNIST. We can confirm biased generated samples, especially '1' or '9'.

Zoom In Zoom Out Reset image size

We iterated this full-MNIST tests for ten times and plotted results. We calculated and evaluated them in the same way of the pair cases. In the figure 7, we plotted only the ratio of '1'. There are two groups, strongly and weakly biased ones. This suggests we have two types of attractor in the case of full-MNIST, along the ratio of generated images classified as '1'. In the figure 8, we show two examples of time series. One is the plot, showing strongly biased case and the other shows a weakly biased case.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

As we observed in the previous section, generated space can be reduced into sub regions, which can cover only the sub-set of the training data set. In the analogical ecology, equations (8) and (9), this type of biased distribution can occur as well, depending on the initial state of the animal niches, a _i or b _i . On the contrary, we know animals often distribute more widely on the field, in the real ecology. If animals distributed on a specific area, they can not eat enough because of decreased amount of foods. This results in decrease of animals around there and they often disperse to elsewhere with enough foods, e.g. spatial Lotka–Volterra system [16, 17]. Based on this analogy, we introduce a kind of population dynamics onto our learning dynamics, equations (3) and (4). Our idea is introducing niche dynamics as the reflection of the food dynamics. The easy way to do that is introducing dynamics for the resource niche, c _i .

When animals stay around a specific resource niche, c _i , in the ecology, the foods around there should be decreased and others should be increased instead. This mechanism can be realized in GAN through the learning dynamics for the discriminator, $D(\boldsymbol{x})$. The discriminator learns the distribution of the training data set, ${\cal X}$. In the training, we use samples of them for the update. In the original GAN, we usually use the uniform sampling weight, w , for the sampling, but we can control the weight appropriately. If we know the all distances, l_ij , between the generated one, ${\boldsymbol{x}}_i$, and the training data point, X _i , the population of the generated samples, P_i , can be evaluated in a straightforward way,

$$\begin{equation} P_i\propto\sum_j\exp(-l_{ij}). \end{equation} \tag{ 19 }$$

This population can be evaluated with just the size of nearest neighbor cluster around each training data. An easy control way for the sampling weight can be written with it,

$$\begin{equation} w_i=\exp\left(-\beta\frac{P_i}{\sum_j P_j}\right), \end{equation} \tag{ 20 }$$

where the parameter, β, is for tuning the scale or gain. The weighting means the sampling is biased to the point with less generated population. For easier computation, we can classify the training data into some representative points, $\boldsymbol{X}_i^*\in{\cal X^*}$, with a clustering algorithm. We can evaluate the generated population distribution with such clustered ones for our cases.

Here, we show the results of controlled MNIST, paired and full one.

As we already confirmed with pair-MNIST, trained generated samples show typically four peaks with strongly or weakly biased ones, in figure 1. In figure 9, we show the results with the controller. In the first one, the control parameter is set as, β = 2.0 and β = 10.0, respectively. With the controller, we want to keep the unbiased state. In other words, we expect the controller stabilizes only two weakly biased peaks among the four ones.

Zoom In Zoom Out Reset image size

At a first glance, we notice the ratio tends to stay around the mid-range. The range covers the weakly biased peaks in the non-controlled results, figure 1. Furthermore, the convergence can be tuned with the parameter, β. When the gain is small, β = 2, the distributions show more diversity in the mid-range. But, we see more convergent ones with the larger gain, β = 10. In the cases with pair-MNIST, we can control the generated distribution.

As the next step, we want to control the generated samples distribution with full MNIST. In figure 10, we show the generated samples with full MNIST. This is calculated with the parameter, β = 10. Those are sampled after the training of 500 epochs. We can confirm more diversified samples than that without the control, figure 6.

Zoom In Zoom Out Reset image size

In figure 10, we can confirm only one converged region. The region lays around the weakly biased one, ${\sim}0.2$, in figure 7. This suggests attractor selection is realized with the controller. However, we could not confirm much less biased distributions, ${\sim}0.1$, which suggests equally covered distribution for all digits.

As we confirmed, we have two peaks in the case of full-MNIST, ${\sim}0.2$ or ${\sim}0.6$. On the other hand, we have only one peak around the area, $0.2 \sim 0.3$, with the controller, figure 11. In the figure 12, we show all trajectories showing the ratio for each digit to check into detail. We can confirm the ratio for '1' is larger than others in both cases. We cannot realize equally covered distributions, unfortunately.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The results, shown here, suggest the simple controller for sampling weight is enough to keep diversity of generated samples with the mechanism of attractor switching. But we should note it is not enough for perfect control for the ideal distribution.