Don’t miss the mismatch: investigating the objective function mismatch for unsupervised representation learning

Many unsupervised representation learning algorithms are based on self-supervised learning [26, 52, 53], which obtains labels directly from data without human annotation to define a pretext task. There are several approaches to self-supervision:

Generation-based self-supervison examines the generation of an arbitrary output from a learned representation of the given input. One line of work improves on autoencoders [51] and variational autoencoders [31] by defining generation-based pretext tasks which lead to representations valuable for required target tasks (e.g., object classification or detection). Examples are denoising [8, 61], colorization [34, 35, 73], or inpainting of images [47, 67]. Recently, a second line of work based on GANs [17] emerged, which adjusts their latent space for representation learning, for example by constraining [50] or changing [14] the architecture. In a third line of research, an autoregressive, transformer-based model achieved state-of-the-art performance on visual representation learning by sequential image generation [20]. Generation-based self-supervison is applied to other modalities as well, e.g., audio [20] or video [57, 62].

Context-based self-supervison recently has moved more and more away from autoencoding data: Early approaches utilize spatial context structure by defining pretext tasks for context generation, like image inpainting [47] or denoising, as a weak form of inpainting [8, 61]. In contrast, approaches for context prediction do not create any image and, for example, try to leverage the knowledge obtained by predicting patch positions [12, 43]. Spatial context can also be encoded by predicting transformations, which has led to a line of research focusing on autoencoding transformations rather than data [16, 37, 49, 72]. Recently, the context-based similarity approach of contrastive learning [19], which utilizes context information between negative and positive pairs, gained popularity and achieved promising results [9, 11, 21, 36, 56]. Contrastive Learning has been linked to mutual information maximization [38, 68], which in ongoing work is used to define pretext tasks through context-based similarity as well [4, 24]. Context similarity by pseudo-labeling through clustering methods is another line of research [7, 71]. Self-supervised relational reasoning combines context-based similarity and context-based structure by discriminating how entities relate to themselves and to other entities and has also been linked to mutual information maximization [46]. Context-based approaches are applied to other data modalities as well, e.g., point clouds [69] or video [15, 28].

Other unsupervised representation learning methods for example combine multiple self-supervised approaches [10, 29, 48, 63], use meta learning [25, 27, 41, 54] or metric learning [6, 65] to learn unsupervised learning rules, or rely on self-organization [1, 58].

Changing the underlying model is one common theme to compare different unsupervised learning techniques [18]. Here, a well known finding is that a larger representation size significantly and consistently increases the quality of the learned visual representations [32].

Varying the amount of data samples has led to interesting observations as well [42, 59]. For example in [3], it is shown that unsupervised learning is capable to learn features of early layers from a single image.

Analyzing self-supervised learning across target domains is another way to define and evaluate benchmarks for unsupervised approaches [8, 42, 64]. Zhai et al. [70] define good representations as those that adapt to diverse, unseen tasks with few examples.

Furthermore, there exist works where the underlying model, the amount of data samples and the target domain is analyzed collectively [18, 44].

Other investigations of unsupervised learning focus on the effect of the multitask pretext learning [13, 55], evaluate the disentanglement of representations [39], investigate the positive effects of unsupervised learning regarding robustness [23], or provide a theoretical analysis of contrastive learning [2, 66].

The objective function mismatch in unsupervised learning is not unknown. Some works directly or indirectly observed that learning a pretext task too long may hurt target task performance, but made no further investigations on this topic [32, 39, 64]. Other works sometimes showed performances of linear target models over training epochs, but did not examine or define the objective function mismatch in detail [58, 70]. Instead, unsupervised multi-task learning and meta learning are proposed as approaches to lower the objective function mismatch [13, 41]. In contrast, this work focuses on defining general protocols to measure mismatches of metrics over the course of pretext task training when a target task is trained on top of the pretext model’s representations. To the best of our knowledge, this has not been done before. Furthermore, we highlight important properties of our evaluation protocols and interesting dependencies of the objective function mismatch.

Naively, we could compare the objective functions of the target and pretext task by using \(\mathrm {M3}\), which we define as the hard objective function mismatch. In most cases, however, the objective functions used to train the pretext model and the target models are not directly comparable. This is due to the usage of different objective functions for both model types, which, i.e., use different (non)linearities. But for some pretext tasks simple, comparable metrics can be defined. These metrics can be used as a proxy to measure the objective function mismatch in a general and comparable manner. A well known example is the accuracy metric, which can be used on the self-supervised tasks of predicting rotations [16] and the state-of-the-art approach of contrastive learning [9]. But comparable metric pairs can not always be found easily. For example, if we train a variational autoencoder and later use its representation for a classification target task, it does not make sense to define a pixel-wise error between the given and generated images as a comparable pretext task metric. To achieve a comparable measurement for this situation, and on the loss curves in general, one could think of individual normalization techniques between objective function pairs. However, we want to be practical and define a measure, which can be used independently of the objective function pairs for every pretext and target model combination. Furthermore, in practice, we might be especially interested in how much the target task mismatches with the pretext task if a mismatch decreases target performance. This is why we define soft versions of our measurements.

Now we can use \({\mathrm {SM3}}\) to measure a soft form of the objective function mismatch on the loss curve obtained by the target models. However, the values of these measurements lie in a range, which depends on the target objective function. Therefore, they are not directly comparable to the measurements on loss curves from other target tasks. This is why we normalize the measurements of the target metric to percentage range and define the objective functions mismatch (\({\mathrm {OFM}}\)) as follows:

The intuition behind this normalization is that we declare \(m_{b}^{T}\) as the value where the pretext model has learned all of the target objective, it was able to learn (with this setting) and \(m_{1}^{T}\) as the value where the model has learned nothing of the target objective. Now we measure with \(\mathrm {OFM}(m^{T}_i)\) for what percentage the learning of a pretext objective hurts the maximum achieved target performance at step \(s_i\). An example is illustrated in Fig. 3. Furthermore, we can normalize the other soft measurements from Eqn. 5 and 6 analogs to Eq. 7.

The \(\mathrm {OFM}\) is a general measure, which can be used for pretext and target models where no good proxy metrics can be defined. With the \(\mathrm {OFM}\) we are able to compare mismatches across different pretext and target task objectives and their combinations. We propose these measurements to obtain quantitative and therefore comparable results for individual pretext tasks. To get the best information about the training process, we encourage to plot the curves formed by our metrics as well. We want to point out that our metrics are not intended to measure target task performance, they measure how much the performance on a target task can decrease when an (ill-posed) pretext task is learned too long. Now, to understand the \(\mathrm {OFM}\) further, we take a look at some cases:

\(\mathrm {OFM}(m^{T}_i)=0\): In this case, solving the pretext task objective did not hurt the performance of the target task objective at this point in training.

\(\mathrm {OFM}(m^{T}_i)=x\): Solving the pretext objective did hurt the performance of the target objective at this point in training by \(x\%\) of what the model has learned. Therefore, we should have stopped training earlier. It is not guaranteed that longer training would hurt performance even more, but a growing \(\mathrm {OFM}\) curve or \(\mathrm {MOFM}\) is a good indicator for that.

\(\mathrm {OFM}(m^{T}_i)>100\) The target objective performance is worse than for the untrained model at this point in training.

\(\mathrm {MOFM}(M^{T})=\infty\) Solving the pretext objective hurts the performance of the target objective from the point of initialization. Because we have learned essentially 0% about the target objective in the training process, there is no interval to be used for normalization. Therefore, we interpret this case as if the model has an infinite mismatch as soon as the model forgets something about the target objective.

For our measurements, we make sure to use metric value pairs from models that do not overfit. We achieve this by applying a convergence criterion on the pretext task and by using the best metric values from each target model evaluation curve. As shown in Appendix C, most observations in our experiments are independent of the use of a convergence criterion, if pretext models are trained long enough and without overfitting. Furthermore, we observe a common behavior in Fig. 1: Target models trained on higher epochs of the pretext model tend to converge faster. This indicates that longer training of the pretext task tends to create easier separable representations which may mismatch with the class label.

To evaluate the stability of our measurements, we show the mismatches of the entire training process and their range (\(+,-\)) using 5-fold cross-validation in Figs. 2 and 3. The range of all other models, we have trained is shown in Appendix C. We observe that \(\mathrm {M3}\) generally seems more stable than \(\mathrm {SM3}\) or the \(\mathrm {OFM}\), since it does not rely so heavily on the target metric values, which can be quite unstable. The instability of the target task mismatch is captured in \(\mathrm {M3}\), but does not matter that much in the overall measurement for most cases. This is favourable if a stable value is desired and unfavourable if one wants to capture the instability of the target task training process explicitly. Furthermore, \(\mathrm {M3}\) is able to compensate target fluctuations with pretext fluctuations. In general, we observe that as long as we calculate the \(\mathrm {OFM}\) across a fair amount of cross-validations (in our case 5), we can make statements about the mismatch. We measure our metrics on the mean losses during 5-fold cross-validation instead of calculating them five times and taking the average. For \(\mathrm {M3}\) both variants are equivalent and for the \(\mathrm {OFM}\) measuring on the mean losses leads to a lower bound in the case where all models converge at step \(s_n\) (see Appendix A for the simple proofs). We prefer to measure our metrics on the mean losses, since this avoids mismatches occurring just in some validation cycles due to small fluctuations of the underlying training procedure. An example is shown in translucent red in Fig. 3 at the beginning of training. We want to point out that the training and validation data differ slightly in every round because of the cross-validation setup. This increases the instability, but shows the general behavior of the metrics for the underlying data distribution. In Appendix C we compare the instability of partially measured mismatches using linear interpolation with mismatches measured for every pretext training epoch and observe a similar instability. However, when using the \(\mathrm {OFM}\) in practice to compare models on a finer scale, we recommend to search for the actual minimal target metric value, since the \(\mathrm {OFM}\) relies on this value at each step. However, when tuning a model for maximum performance, one searches for this value. Thereby looking at the \(\mathrm {OFM}\) curve gives good indications in which interval one should search. This makes this protocol useful for performance tuning, if enough computational power is available.

We hypothesize that large representation sizes tend to lower the \(\mathrm {OFM}\), which could be one reason why representation sizes are large in unsupervised learning. To affirm this hypothesis empirically, we train our pretext models with varying representation sizes on different target tasks while fixing all other model parameters. Figure 4 and Table 1 show that the \(\mathrm {OFM}\) tends to decrease when we enlarge the representation size. A reason for that might be that target models can exploit the high dimensional space of large representations to find better fitting clusters for their target task. We found an exception of this behavior, where we use larger representations of SCLCAE for the easy task of object hue prediction. Here, the target models trained on the untrained pretext models with larger representation sizes already achieve high performance due to a larger number of color-selective, random features. Further learning of the pretext model, in this case, does not lead to a high-performance gain and forgetting these sensitive random features during training leads to a high mismatch. Additionally, we observe that mismatches decrease, when we decrease the representation size for generation-based methods. A reason could be that the pretext models are forced to generalize to solve the target task for small representation sizes due to the limited amount of features in the bottleneck, or simply underfit on the pretext task.

In Fig. 4 and Table 1 we observe a \(\mathrm {OFM}\) spike early in training for more complex target models. This spike occurs probably because nonlinear target models make better sense of specific random features at pretext task initialization, in contrast to the linear target model. Besides early spikes, mismatches tend to decrease when we add complexity to the target model. A model with increased nonlinearity has more freedom to disentangle representations, which do not fit properly with the target task. Again we found an exception where the \(\mathrm {MOFM}\) is lower for linear models when predicting the object hue after contrastive learning which can be appointed to the color-selective, random features of the untrained pretext model.

We vary the augmentations used for the pretext and target model by removing the color jitter and the image flip from our base augmentations successively. Figure 4 shows that augmentations can have a positive or negative impact on the mismatch. E.g., when predicting the object hue, the ill-posed color jitter augmentation increases the mismatch significantly.

Here we use our metrics to examine findings stated in [13, 32] and [64, 70], where it is argued that some pretext tasks are better suited for different target tasks. We fix the underlying data distribution by using the 3dshapes dataset and train our target models for the different tasks. These tasks require a generic understanding of the scene like coarse-grained knowledge about object type and hue and fine-grained knowledge about shapes, positions and scales. In Fig. 5 and Table 2, we observe that pretext models tend to learn pretext task specific features and discard features that are not needed to solve the pretext task during training. These models, therefore, mismatch with ill-posed target tasks. For example, rotation prediction discards features corresponding to the hue while it learns much about the orientation of the object.

In Fig. 6, we apply our metrics to ResNet models for several pretext and target tasks. For contrastive learning we observe a small \(\mathrm {OFM}\) for Cifar10 and Cifar100, which occurs late in training after pretext model convergence. However, when we use contrastive learning as pretext task for fine-grained tumor detection on the PCam dataset, we observe a mismatch before pretext model convergence. For the well-known rotation prediction pretext task, we oberved a high mismatch on Cifar10 classification early in pretext training. In Table 3, we show the corresponding mismatches measured until pretext model convergence.