RII-GAN: Multi-scaled Aligning-Based Reversed Image Interaction Network for Text-to-Image Synthesis

Reed et al. [22] first demonstrated that conditional GAN (cGAN) [15] can synthesize credible images from text descriptions. To generate the \(256\times 256\) resolution images, Zhang et al. [36] developed a stacked structure GAN (StackGAN) by several generator-discriminator pairs. Xu et al. [32] extended it and introduced cross-modal spatial attention to calculating the attention matrix for text-to-image fusion. Zhu et al. [41] explored a dynamic memory scheme for improving T2I fusion process. Li et al. [10] introduced channel attention to exploring the relationship between the image feature channel and word context vector. To improve the semantic consistency between the generated images and text, Yin et al. [35], and Tan et al. [27] utilized Siamese structures with contrastive learning loss to enhance semantic consistency. MirrorGAN [19] leverages an additional inverse inference model to caption the generated images, enhancing the generator’s ability to create semantically consistent images by capturing visual-textual consistency. However, it requires pre-training of the I2T model. Due to its substantial size and complexity, the method is relatively challenging in training and potentially less efficient than end-to-end models. To improve the generation quality, Yang et al. [33] explored the semantic common between different sentences describing the same image by utilizing multiple single-sentence generation and multi-sentence discrimination (SGMD) modules. Considering the properties of image complexity and text generality, Tan et al. [28] designed a regularized GAN framework to distinguish the critical and unimportant information of generated images, making the network more inclined to focus on the necessary semantic parts of the feature map.

Recently, some works have simplified the stack structure into a single generator-discriminator pair. For example, Zhang et al. [38] chose the one-stage generative structure as the backbone for XCM-GAN through five contrastive learning pairs between image, sentence, and word embeddings. Tao et al. [29] also chose a single-stage GAN-based T2I model to achieve text-image fusion. They employed sentence embedding as conditional input, making fusion more effective. For making the model pay more attention to semantic focus areas, inspired by [29], Liao et al. [12] introduced a weakly-supervised mask prediction for the fusion module. In [11], Li et al. provided the image feature manifold structure directly to the generator by a memory bank, which previously stores the authentic image feature maps. Unlike them, the proposed RII-GAN improves the quality of generated images by exploiting a combination of a simple single-stage generation structure and a new reversed image interaction mechanism.

The early T2I model achieved text-to-image fusion by directly concatenating text and image feature maps. Subsequently, some works such as AttnGAN [32] and StackGAN++ [37] used the cross-modal attention mechanism to improve the fusion effect. However, the introduction of an attention module inevitably increases computational consumption. To solve this problem, Yin et al. [35] introduced condition batch normalization (CBN) into text-image fusion. It fuses text and image features by disentangling text embedding and performing the affine transformation. Afterward, Tao et al. [29] proposed a deep fusion module, which achieves an excellent result by adopting sentence embedding to achieve multi-scale fusion. Liao et al. [12] used masks to pay more attention to the generation of crucial feature regions, thus improving the fusion effect. Ye et al. [34] found that mutually independent fusion modules conflict with each other during training. To address this issue, they use a bidirectional long short-term memory network (BiLSTM) [25] to establish a long-term dependency on the independent fusion blocks, which helps to reduce the difficulty of model training. Our framework introduces an adaptive affine transformation, which also provides a solution to achieve long-term dependency of fusion blocks while implementing textual information augmentation by adaptively considering the feature maps computed forward by the current generator.

Image-text and image-image feature alignment schemes are the research focus of T2I generation tasks. Xu et al. [32] utilized BiLSTM [25] and Inception-v3 model [26] to build the deep attentional multimodal similarity model (DAMSM), which is applied to vary text embedding and image features to the same dimension and perform feature alignment to improve the image-text consistency. To reduce the randomness of the generated images, Li et al. [10] developed perceptual loss to align the features of the generated and authentic images. Yin et al. [35] found that the T2I model based on GAN is challenging to distinguish the subtle differences between each representation of the same image. Therefore, they use Siamese structure to align different generated images with corresponding text descriptions but the same reference object. Zhang et al. [38] improved their work using five pairs of contrastive learning. Afterward, Tan et al. [28] argued that the crucial to generating high-quality images is whether the generator can identify critical features in the forward calculation process. Therefore, they used the semantic disentangling module (SDM) to disentangle the generated foreground image features to perform feature alignment with real image features. We propose a novel RIIN for multi-scale image feature alignment with the generator. It can provide rich image distribution information to the generator while enhancing semantic consistency and mitigating mode collapse.

The optimization goal of GAN is to achieve the Nash equilibrium of the generator and discriminator, which usually leads to mode collapse due to unstable training. Although some methods [1, 2, 20] attempt to improve the stability of GAN training, few reports focus on modifying network architecture. To maximize the prior knowledge of text-image training pairs and let T2I models eliminate the dilemma that lacks the learning opportunities of image feature manifold structure, we propose the reversed image interaction network in this work. This study has changed the basic structure of the generator-discriminator pair in traditional GAN by embedding the proposed RIIN so that the RIIN can indirectly provide actual image prior knowledge. RIIN incorporates a custom-designed convolutional network, which synchronously shapes with the generator’s output during the forward computation process. This unique feature allows modification of the output value of the intermediate feature map via affine transformation. The alignment feature in our model is domain-agnostic, focusing instead on the intermediate domains between text and image throughout the image generation process. RIIN will update the generator by reversed transformation for cooperation with the GAN mechanism. Thus, during the training process, RIIN can provide a stable image manifold structure based on the capabilities of the improved generator and discriminator. If the effect of RIIN is only to provide image feature manifolds, pre-training models with fixed parameters migrated from some large datasets, such as ResNet-101 [6], ViT-L/16 [4], can improve the ability to extract image feature manifolds at the early stage of training. However, the large pre-trained model is built on a single image modality. If the generation network features are forced to align with those of the pre-trained model, the results may not be satisfactory. Since the T2I model generator involves cross-modal transformation, there must be an intermediate domain in the transformation process from the text domain to the image domain. The intermediate domain is a potential spatial domain with text and image manifolds but does not favor either side. In the training stage, the intermediate domain cannot be controlled through external supervision. The intermediate domain of T2I generation is part of the latent space learned by GAN. This is the main reason many works use the GAN mechanism for implicit learning of image distributions. Monitoring the learning results of the intermediate domain is challenging because we need valid labels and metrics. Therefore, traditional GANs are prone to mode collapse, gradient disappearance, etc. Fortunately, in the training phase, we can easily access three information sources related to the intermediate domain: random noise distribution, text distribution, and actual image distribution. With the help of the GAN mechanism, we can obtain weakly-supervised labels from the proposed RIIN for intermediate domain learning by effectively using these three information sources. We develop the CBNs-based Affine Block in RIIN as shown in Fig. 2. It aims to make the network adaptively close to the intermediate domain representation and provide image information of feature maps from large to small. Moreover, this is a cross-mode task, the information contained in the generator’s feature map at different resolutions should be different. A larger feature map (\(128\times 128\)) carries more image information. Correspondingly, a smaller feature map (\(8\times 8 \)) takes more text information and noise distribution. The generator and RIIN are mutually constrained and optimized for each other.

In the proposed RIIN, the actual image \(\hat{ {I} }\)(\(256\times 256\)) is sent to a convolution layer to obtain a \(256\times 256\) feature map \(h_0=conv(\hat{I}\)). \(h_0\) will then be conveyed to five Down Blocks in turn. To quickly approach the intermediate domain features in the generator and make better use of the real image prior, as shown in the lower right corner of Fig. 2, each Down Block has four inputs: the output feature \( h _ r \) from the upper Down Block, the real image \(\hat{ {I} ^{M \times M}}\), the sentence vector s, and the noise distribution z. Because we want to approach the implicit space with text feature, image feature, and input random noise distribution, this study considers incorporating all these three information source feature components and one upper-level feature into Down Block. The affine layer is a CBN framework inspired by [29], which takes the sentence vector s as input and fuses the textual information into the feature map. Several parameters of CBN are defined as follows.where \( {\gamma _r} \in {{ {\mathbb {R}} }^{{N \times C_{in}}}} \) is the channel-wise scaling parameter, \({\beta _r} \in {{ {\mathbb {R}} }^{{N \times C_{in}}}}\) is the shifting parameter. \( {h'_r} \in {{ {\mathbb {R}} }^{N \times {C_{in}} \times H \times W}} \) and \( {{\hat{h}}_r} \in {{ {\mathbb {R}} }^{N \times {C_{in}} \times H \times W}} \) are the input and output of the affine layer. \( {\hat{h}}_r \) will be used to concatenate with the real image feature map to obtain the final Down Block output feature \( {h_r} \in {{ {\mathbb {R}} }^{N \times {C_{out}} \times H \times W}} \). The dimensions of RIIN output features are respectively \(8 \times 8\), \(32\times 32\) and \(128\times 128 \), and these features will align with generator features of the same size. The feature alignment loss computes the mean square error between features for shortening the feature distance. This loss is used to establish a bi-directional constraint between the generator and RIIN, which is defined bywhere \( h_r^{M \times M} \) and \( h_g^{M \times M} \) represent the \( M \times M \) feature map of the RIIN and the generator, respectively. \({\Vert \cdot \Vert _2}\) represents the mean square error loss. Later experiments will discuss the impact of two hyper-parameters \(\lambda _1 \) and \(\lambda _2\).

According to [34], the isolated fusion modules conflict with each other, increasing the training difficulty. Therefore, this study designs an adaptive affine-based generator consisting of six UP Blocks and one UP Block0. We developed two AABs for each UP Block. When the proposed adaptive affine-based generator performs forward computation, the text embedding will be updated by the Adaptive Block in the UP Block (Fig. 2), and input into the subsequent affine layers. Figure 3 shows the architecture of Adaptive Block. This adaptive structure links isolated affine layers to the network backbone. It provides the affine transformation with the path to obtain the current layer feature and establishes long-term dependence on other network modules to reduce the training difficulty. The information enhancement of the AAB is mainly completed in the Adaptive Block. As shown in Fig. 3, the Adaptive Block has two inputs: the current layer feature map \( h_{g}^{{'M} \times M} \) and the sentence vector s. The convolution kernel size is the same as the dimension of the feature map. \(h_{g}^{'1} \in {{{\mathbb {R}} }^{N\times 100}} \) is the depth-wise convolution transformation output. The semantic condition \({s_{con}} \in {{ {\mathbb {R}} }^{N \times 356}} \) is the result of concatenation between \( h_{g}^{'1} \) and s, which will convey into the multilayer perceptron (MLP). We use Tanh as the final activation function to compute an attention map within values in (-1, 1). Finally, we adopt a residual structure to acquire the adaptive information-enhancing sentence vector \(s'\) using\(s'\) then is used to conduct the affine transformation. The channel-wise scaling parameter \({\gamma _g} \in {{ {\mathbb {R}} }^{{N\times C_{in}}}}\) and the shifting parameter \({\beta _g} \in {{ {\mathbb {R}} }^{{N\times C_{in}}}}\) are defined as followswhere \( {h'_g} \in {{ {\mathbb {R}} }^{N \times {C_{in}} \times H \times W}} \) and \( {{\hat{h}}_g} \in {{ {\mathbb {R}} }^{N \times {C_{in}} \times H \times W}} \) are the input and output of affine layer, respectively.

This study proposes a dual-channel feature alignment discriminator to enhance the semantic consistency between generated images and text descriptions. Since the major role of the discriminator in the T2I task is to capture the feature matching degree between the information of text and image feature map, the model should perform internal unwrapping and alignment operations on image and text features. The problem of extracting the matching information of two modes should be related. For example, extracting matched text information with fixed image features inevitably introduces the independent semantic part (redundant data) of image features. And extracting image-related features in the case of fixed text embedding will also have similar problems. Therefore, the extraction of matching information of the two modes should be carried out jointly to consider each other. Yin et al. [35] utilized Siamese structure and contrastive loss to forcibly improve the matching degree of image feature and text embedding features. Their strategy consumed enormous computing resources. Therefore, this study designs a new discriminator framework to improve the disentangling capabilities of the bi-modal information embedding of image and text. In the proposed discriminator, we need image and text feature information that can match each other. Different objects need to match different key information, which leads to the discriminator in the feature disentangling obtain some unmatched features if without feature selecting performance. So, feature alignment is required to guide feature extraction. Each channel of the dual-channel consists of the disentangling feature of one mode (text/image) and the original feature of the other (image/text). To improve the ability of the model to select correctly matched text and image features, we perform feature alignment constraint operations on the final disentangling fused features in two channels, as shown in Fig. 4.

The structure of the proposed DFAD is shown in Fig. 4. Firstly, one obtains the initial image feature map P through a series of Residual Down blocks. The P and the extended sentence embedding \( S^{8 \times 8} \) are concatenated and sent respectively to the multilayer perceptron of two different channels (MPL1 and MPL2) to obtain the upper channel’s sentence attention map \(M_S\) and the lower channel’s image spatial attention map \(M_I\). We utilize \(M_S\) and \(M_I\) to obtain two disentangling feature maps \( {\hat{h}}_{d\_image}^{8 \times 8} \) and \( {\hat{h}}_{d\_text}^{8 \times 8} \) usingThen, we respectively send the concatenation between \( {\hat{h}}_{d\_image}^{8 \times 8} \) and \( S^{8 \times 8} \), \( {\hat{h}}_{d\_text}^{8 \times 8} \) and P to two different residual blocks to obtain the feature map \( h_{d\_{f_1}}^{4 \times 4} \) and \( h_{d\_{f_2}}^{4 \times 4} \). These feature maps are used to align features. This study uses the mean square error loss \( {\mathcal{L}_{{D_f}}} \) to represent the feature distance.Since the discriminator adopts a dual-channel structure, the result of the forward computation is the average of the dual-channel output, as shown in Fig. 4. This study uses the adversarial hinge loss \( {\mathcal{L}_{{D_{adv}}}} \) [13] and MA-GP loss \( {\mathcal{L}_{{D_{MA - GP}}}} \) [29] for the proposed discriminator.Generator adversarial loss \( {\mathcal{L}_{{G_{adv}}}} \) is the cross-entropy loss defined byFinally, we obtain the loss functions of discriminator, generator, and RIIN as follows:To better understand the training process, we present the implementation details of the proposed RII-GAN framework with the following pseudo-code.

We conduct all experiments on two standard T2I datasets: CUB-Bird [31] and MS-COCO [14]. The former has 2,933 test birds (50 categories) and 8,855 training birds (150 categories). Each bird has ten English sentences to describe the fine-grained visual scenes. The latter consists of 82,783 training images and 40,504 test images. Each image has five sentence annotations. Compared with CUB-Bird dataset, images in MS-COCO present complex visual scenes, which make the T2I generation tasks more challenging.

We use three broadly utilized metrics CLIPScore (CS) [7], Inception Score (IS) [24], and Fréchet Inception Distance (FID) [8] to quantify the performance of all approaches in terms of image-text alignment and image quality.

Image-text alignment: CS adopts a pre-trained CLIP model [21] to map image and text into the same feature space and calculates the cosine similarity between the text feature and image feature. A larger CS indicates that the generated image has a more significant semantic similarity to the text.

Our proposed model RII-GAN is implemented using the PyTorch toolbox. The model training consists of two distinct phases. In the initial phase, the two independent BiLSTMs [21] are trained as text encoders, leveraging the DAMSM approach for computing image-text similarity [2]. Specifically, we employ DAMSM to train a pair of text and image encoders that can align with each other, thereby obtaining an efficient text encoder. Each BiLSTM is trained using CUB-Bird and MS-COCO datasets on a single RTX 3090 GPU. The BiLSTM outputs global sentence vectors with 256 dimensions used as text embeddings for the subsequent generator, RIIN, and the discriminator. In the second phase, we freeze the text encoder and train the GAN. We assign a batch size of 32 for both CUB-Bird and MS-COCO datasets. The generator and RIIN in our framework are simultaneously trained using Adam optimizer [9], with parameters \(\beta _1\) and \(\beta _2\) set to 0.5 and 0.999, respectively. The optimizer is initiated with a learning rate \(2e-4\), which starts to linearly decays over the final one-third of the training epochs approaching \(5e-5\). In parallel, the discriminator also employs the Adam optimizer with \(\beta _1\) and \(\beta _2\) set to 0.5 and 0.999, respectively and a fixed learning rate of \(4e-4\). Throughout the second phase, RIIN training status is synchronized with the generator and alternately trained with the discriminator. The hyperparameters \({\lambda _1}\) and \({\lambda _2}\) are respectively set to 2 and 3. The subsequent experimental sections present a comprehensive discussion of the choice of these parameters. The model undergoes training for 600 epochs on CUB-Bird dataset and 200 on MS-COCO dataset. The image generation process only retains the generator component during the testing phase.

In this study, \( \lambda _1 \) and \( \lambda _2 \) are two important hyper-parameters in the feature alignment loss \( \mathcal{L}_{{G_f}} \) of the generator and RIIN. According to (2), these parameters will guide the generator and RIIN to pay more attention to the feature corresponding to the larger parameter. Figure 5 shows the FID scores in the last 200 training epochs on CUB-Bird dataset with different \(\lambda _1\) and \(\lambda _2\). The first row evaluates the effect of \(\lambda _1\) when \(\lambda _2\) is fixed as 3, and the second row investigates the impact of \(\lambda _2\) when \(\lambda _1\) is fixed as 2. The best FID score with different parameters is marked with red numerals. Empirically, we found from the first and the second rows of Fig. 5 that when \(\lambda _1=1,1.5,2.5,3 (\lambda _2=3)\) or \(\lambda _2=2,2.5,3.5,4 (\lambda _1=2)\), the FID scores are unstable and fluctuates obviously with the increase of epochs. Figure 5j and k summarize the optimal and average FID scores obtained in the last 200 training epochs when \(\lambda _1\) and \(\lambda _2\) take different values. We observe that when \(\lambda _1=2\), \(\lambda _2=3\), our model’s average, and optimal FID scores reach the bottom, indicating that the network reaches a local minimum. As shown in Fig. 5i, taking \(\lambda _1=2\), \(\lambda _2=3\), the model obtains the minimum FID score under relative stability, especially in the last 100 epochs. Therefore, parameters \(\lambda _1\) and \(\lambda _2\) are set to 2 and 3 in all experiments.

Table 1 shows the quantitative results of our RII-GAN and several advanced T2I models on CUB-Bird and MS-COCO datasets. The results of these T2I models on two datasets come from their publicly available codes on the web. This table shows that our RII-GAN obtains the third-highest CS, lower than the baseline DM-GAN. The DM-GAN achieves the best CS (71.91), indicating better text-image consistency maintenance on CUB-Bird dataset. Our RII-GAN on CUB-Bird dataset achieves the lowest FID scores (12.94) and the highest IS (5.41). These results show that the images generated by our RII-GAN are closer to the actual image distribution in terms of maintaining image quality. On the large-scale MS-COCO dataset, our RII-GAN also achieves competitive results. The statistics in the second last column in Table 1 show that our RII-GAN achieves the lowest FID score of 19.01 among current state-of-art methods, which indicates the ability to generate more realistic images. The CS of our RII-GAN is much higher than that of AttnGAN, DM-GAN, and DAE-GAN and is almost on the same level as the state-of-the-art DF-GAN. The score is relatively much better than the CS scores in CUB-Bird dataset. It illustrates that our RII-GAN is much superior to AttnGAN, DM-GAN, and DAE-GAN in terms of semantic consistency on large datasets with the help of RIIN, AAB, and DFAD. Three metrics on both datasets demonstrate that our RII-GAN can generate higher-quality images.

In this section, we visualized some synthesized images by our RII-GAN and three other advanced models: AttnGAN [32], DM-GAN [41], and DF-GAN [29] on CUB-Bird and MS-COCO datasets shown in Fig. 6. AttnGAN and DM-GAN are two classical multi-stage generation methods, and DF-GAN and our RII-GAN are one-stage generation methods. The images in each column are generated by different approaches based on the scene’s text at the top of each column. These text descriptions are randomly selected from datasets. According to semantic consistency, our method captures more text detail descriptions. As shown in Fig. 6, in the first column, our RII-GAN captures the “yellow on its tips of its wings” detail in “green color with yellow on the tips of its wings” and reflects it in the generated image, which fails in the other methods. In the third column, the generated images of AttnGAN, DMGAN, and DF-GAN show a lot of overlap between the water scene and the bird body. By contrast, the images generated by our RII-GAN method can reflect the details and colors of birds on the lake. For a more challenging MS-COCO dataset, it can be observed that our RII-GAN can still generate images of the complex scene, e.g., RII-GAN successfully focuses on the “clock tower” keyword information. In contrast, AttnGAN, DMGAN, and DF-GAN only generated objects with inaccurate shapes through the keyword “tower” (the fifth column). In columns 6 and 7, the “bulls” and “zebras” are visible in our method but hard to recognize in others. As shown in the 8th column, the proposed model better reflects the meaning of the text, such as “a group of people”. Generally, these subjective visual comparison results confirm the effectiveness of the proposed RII-GAN.

Since there is no adequate convincing criterion to judge the consistency of image-text semantics of the generated image, we conducted a human test shown in Fig. 7. The experiment randomly selected 100 text descriptions from CUB-Bird dataset and then invited 30 volunteers to compare the results of our RII-GAN and two popular baselines AttnGAN [32] and DF-GAN [29]. In this test, the user receives three images generated from three models and a corresponding text description. Each user sorts these images according to realism and text alignment. As summarized in Fig. 7, our RII-GAN achieved the highest ranking of \(50.46\%\) and \(63.08\%\), respectively, regarding text alignment and realism. The conclusion also shows that our RII-GAN has obtained better approval in human perception.

Visualization by t-SNE analysis. To investigate the feature distribution between the RIIN and the proposed adaptive affine-based generator, we utilize the t-SNE algorithm to visualize the feature alignment between the generator (blue triangles) and our RIIN (orange circles) on CUB-Bird dataset. Figure 8a shows the two-dimensional distribution of noise \(z\sim N(0,1)\) and real images before forward computing. Figures 8b, c, d show feature distributions of different scales. We found that the feature distance between the generator and RIIN is large in the initial stage. However, as the forward computing of RIIN gets deep into the reversed image interaction process (from \( 128 \times 128 \) to \( 8 \times 8 \)). The feature distributions of both modules successfully converge on the same intermediate domain (see Figs. 8c, d).

Image diversity The randomness of the noise z controls the diversity of the generated images. z is sampled from a Gaussian distribution N(0, 1). To explore the influence of z on the generated images, we conducted control-variable experiments on the text input and random noise z of our RII-GAN, as shown in Fig. 9. The random noise z is fixed at the same level. The sentence embedding of the input model varies by changing the specific words in the text description. From the generation images shown in Fig. 9, we found when the color attribute in the description changes, the images generated by RII-GAN can keep other elements except for the color unchanged (such as the bird’s pose and background information). This result demonstrates the high controllability of our method for generating images. If the text embedding is fixed under the same vertical line, changing the random sampling noise z, we found that different z significantly impacts the object pose and background of the generated image. By varying the input z, the generator synthesizes bird images with different angles and backgrounds. Therefore, the results of Fig. 9 demonstrate that our RII-GAN can effectively disentangle the attribute of the input text description and control the modeling of the sample relevant regions while not affecting the generation of non-relevant regions during the disentanglement process. Figure 10 presents more examples of images synthesized by the proposed RII-GAN on CUB-Bird dataset.

Impact of RIIN, DFAD and AAB. The proposed framework comprises three main modules: AAB, RIIN, and DFAD. We carried out ablation experiments to validate the impact of each module in our RII-GAN on CUB-Bird dataset. The baseline model is a one-stage GAN-based T2I network. The evaluation metrics are FID and IS summarized in Table 2. The role of AAB is to use the current feature layer to adaptively provide a basis for updating text embedding. Without (w/o) AAB, the FID and IS scores are 13.25 and 5.37±0.03 on CUB-Bird dataset while the FID value on MS-COCO dataset is 21.23. It verifies that adding AAB into the framework helps generate more realistic images. We also investigate our model with (w/) AAB and RIIN, and without DFAD, we find the FID increases from 12.94 to 14.03, and IS decays from 5.41 to 5.39 on CUB-Bird dataset and the FID increases from 19.01 to 20.03 on MS-COCO dataset. It can be concluded from the table that the use of the dual-channel feature alignment structure is beneficial for semantic extraction and matching of the discriminator. Finally, we also evaluate the effect of the RIIN module. When RIIN is removed from our model, the FID has significantly increased by 9.89%, and IS has decreased by 2.96% on CUB-Bird dataset. On the other hand, the FID increased substantially by 11.68% on MS-COCO dataset. The results in Table 2 reflect that the proposed RIIN has a more significant impact on the performance of our framework than AAB and DFAD.

In Fig. 11, we show the generated images of the Baseline (Base.), RII-GAN w/o RIIN, RII-GAN w/o DFAD, RII-GAN w/o AAB, and RII-GAN on CUB-Bird dataset ((a)-(e)) and MS-COCO dataset((f)-(j)). As shown in Fig. 11, our RII-GAN performs the best in image generation quality and semantic consistency, which is more evident on CUB-Bird dataset. By ablating each module of AAB, DFAD, and RIIN, it can be observed that model w/o RIIN has the greatest effect on the quality of the generated images (eg. (b), (c), (j) of Fig. 11). This is because RIIN directly interacts with the generator to learn the real image feature manifold. In detail, it reversely encodes the features of the real image, and they are aligned with the features forward encoded by the generator. Besides, AAB and DFAD also significantly contribute to the model’s ability to generate images that are more realistic and richer in semantic detail (“tomatoes, egg” in (g), “forks” in (i), and “lot” in (j) of Fig. 11) and object layout (“water tower” in (h) of Fig. 11). Specifically, AAB is used to adaptively update the text embedding, which facilitates the affine module to capture the text information acquired by the generator, thereby enriching the semantic detail of the model. Additionally, the dual-channel structure of DFAD enhances the robustness of the discrimination process as it more effectively identifies the critical matched semantics of text and image and dismisses redundant information.

The dual-channel feature selection scheme of DFAD. To further explore the impact of dual-channel feature selection and feature alignment of DFAD on the generated images, we conducted the experiments shown in Table 3. When the alignment structure of the discriminator is removed, the FID score increases from 12.94 to 15.92 with no change in IS. This result demonstrates that the feature alignment process significantly affects the quality of generated images. We also found that removing different modal feature channels (image channel/text channel) has a significant impact on the model’s performance. As indicated in Table 3, the FID score increases by 0.63, and the IS decreases by 0.11 (compared to our DFAD in the first row) if only the image channel is removed. The FID score increases by 3.97, but the IS increases by 0.06 after removing the text channel. The experimental results demonstrate that the joint action of the dual-channel structure and feature alignment work together to achieve better results.

Multi-scale feature alignment mechanism of RIIN. In addition, we investigated the effect of the multi-scale feature alignment scheme in RIIN. We carried out different combinations of feature alignment and summarized the results in Table 4. Row 2-4 represents the absence of a branch in RIIN. According to comparison results, we found that the effect of feature alignment in terms of optimization capability can increase sequentially with increasing spatial resolutions. This phenomenon demonstrates that the reversed interaction process in large resolutions helps the network to generate more realistic images.

The configuration and quantities of AAB in UP Blocks. Then, as presented in Table 5, we evaluate the impact of different numbers of AABs in RII-GAN on CUB-Bird dataset. Column ‘stages’ in Table 5 corresponds to the number of UP Blocks with AABs, where in this experiment, we comprise two AABs in each UP Block as shown in Fig. 2. UP Blocks are accumulated in ascending resolution order and so does the AABs in each UP Block. The generator needs 6 UP Blocks as images would need to be generated from \(4\times 4\) to \(256\times 256\). Hence, we conduct experiments for stages 1 to 6. For example, the sub-experiment for stage 5 has only the first 5 UP Blocks containing AABs. It can be observed that as the quantity of UP Blocks that contain AABs increases, there is a consequent decrease in the model’s FID and an increase in the IS. This trend indicates a continuous enhancement in model performance by an increased number of AABs up to the maximum, which proves its effectiveness.

Another experiment is conducted to evaluate the number of AABs in each UP Block of the generator, as presented in Table 6. Controlling the total number of affine blocks as 4 in each UP Block, which already ensures a sufficient text-to-image fusion [34] [29], we change the number of AABs while appropriately allocating base affine blocks to figure out the optimal configuration. The experimental models are i) one AAB within each UP Block, which consists of four base affine blocks; ii) two AABs within each UP Block which consists of two base affine blocks; iii) three AABs within each UP Block, where the first two AABs incorporate single base affine block and the last AAB consists of two base affine blocks; iv) four AABs within each UP Block, each incorporating one base affine. From the experimental results, we observed that combining two AABs in each UP Block gives the optimal quantitative results for both FID and IS. FID of the optimal configuration reaches 12.94, slightly lower than the second-best 13.13 of three AABs, IS scores of the other three models are approximately at a similar level, while the chosen model is relatively high, reaching 5.41. This is mainly because this architecture potentially establishes a balance capable of preserving model depth while avoiding excessive complexity in the network.