Classification of incunable glyphs and out-of-distribution detection with joint energy-based models

Detecting and localizing glyphs or symbols on historical prints for reproduction or fine-grained analysis is mostly coupled to the more general efforts and algorithmic developments in OCR [21]. Early approaches typically rely on geometric rule-sets [14], classical shape descriptors [48], or carefully crafted features in a machine learning approach [73] to locate and categorize recurring symbols. Subsequent methods use different variants of template matching [6, 7, 9, 25], employ graph matching [13], or follow the advances in deep learning with a particular focus on few-shot reference learning [61], object detection [8], or probabilistic generative modeling for generalized glyph shape assessment [26].

The concept of energy-based models (EBM) dates back to early research on the collective properties of neural networks [35] and Boltzmann machines [1] before being set forth as a distinct approach by LeCun et al. [41]. Since then, EBMs have become an essential member of the family of deep probabilistic and generative networks [4]. They are used for implicit generative modeling [20, 54], predictive modeling [28], continual learning [44], iterative reasoning [18], compositional learning [16, 17], and OOD detection [22, 65]. They also have been used to re-interpret the workings of other approaches such as Variational Autoencoders [69] or Generative Adversarial Networks [11]. We refer the reader to the very instructive works by Du et al. [20], Song et al. [60], Osogami [55], and Arbel et al. [2].

OOD samples typically stem from a secondary distribution that differs from ID data by a semantic shift in the label space or a covariate shift. Yang et al. [71] provide a detailed review on OOD detection methods and their commonality with open set/open world recognition [39]. In the context of predictive modeling, popular OOD detection directly rely on the softmax confidence scores [31, 45], evaluate the distance of the sample to the nearest class-specific distribution [42], or utilize energy scores computed on the classifier’s logits [46]. Recent methods build upon the observation that ID data tends to show higher vector norms in gradient space compared to OOD data [36], or evaluate the k-nearest neighbor distances between feature vectors of the training set and an inference sample [63]. Furthermore, some early works explore the idea of the typical set for OOD detection [12, 50].

Given a limited number of observations \({\textbf{x}} \in {\mathbb {R}}^{D}\), we often want to explain the underlying distribution of that data by constructing an estimate \(q_{\theta }({\textbf{x}})\) of the true probability density function \(p({\textbf{x}})\), ideally using as few assumptions about the data as possible.

Energy-based models seek to establish a relation between these probabilities and certain energy states using a parameterized energy function [41]. This relation is expressed as a Boltzmann distribution \(q_{\theta }({\textbf{x}})= \frac{\exp \left( -E_{\theta }({\textbf{x}})\right) }{Z_{\theta }}\), where \(E_{\theta }({\textbf{x}}): {\mathbb {R}}^D\mapsto {\mathbb {R}}\) is the energy function with parameters \(\theta \) which maps a point of a D-dimensional input space to a single scalar state, called the energy [37]. The partition function \(Z_{\theta } = \int _{{\textbf{x}}}\exp \left( -E_{\theta }({\textbf{x}})\right) d{\textbf{x}}\) normalizes the energy magnitudes with respect to \({\textbf{x}}\) to ensure that the fundamental properties of a probability distribution are satisfied. If the parameters \(\theta \) are configured well, realistic observations are assigned low energy values, and unlikely observations correspond to high-energy states. Intriguingly, the choice for \(E_{\theta }({\textbf{x}})\) places no other restrictions on the function type and can be efficiently parameterized by a neural network with weights \(\theta \).

For most scenarios, optimizing these weights with conventional maximum likelihood learning on some training data \(\mathcal {X}_{\text {train}}\) is not straight-forward as evaluating the normalization constant \(Z_{\theta }\) (which is required to assess the likelihood of \({\textbf{x}}\)) is either analytically intractable or requires time-consuming numerical integration. On the other hand, only estimating unnormalized densities by maximizing \(\exp \left( -E_{\theta }({\textbf{x}})\right) \) is not feasible since the likelihood of the training data changes with different values for \(Z_{\theta }\).

Fortunately, even though we cannot estimate the normalized densities directly, we can use Contrastive Divergence (CD) to approximate the gradients of the energy function [10, 33, 34, 67]. More formally, the partial derivative of the negative log-likelihood objective \(-\log q_{\theta }({\textbf{x}})\) with respect to weights \(\theta \) can be expressed asThis contrastive update scheme increases the likelihood of samples from the true distribution \(p({\textbf{x}})\) while reducing the likelihood of samples from the generated distribution \(q_\theta (x)\), eventually reaching an optimum at the equilibrium \(p({\textbf{x}})=q_{\theta }({\textbf{x}})\).

Now, instead of evaluating the integration of \(Z(\theta )\), we approximate it numerically by taking a sample from the model distribution \(q_{\theta }({\textbf{x}})\). Although we are still not able to analytically sample from \(q_{\theta }({\textbf{x}})\), we can set up a Markov Chain that stochastically explores the model distribution and eventually converges to the equilibrium/steady-state distribution \(p({\textbf{x}})\) using \(K\) cycles of Monte Carlo sampling (MCMC). This iterative transition exploits the ratio between two evaluated points \(q_{\theta }({\textbf{x}})\) and \(q_{\theta }(\mathbf {x^{\prime }})\), which allows to cancel out the partition function (cf. [68]).

To reduce very long mixing times of the Markov chain, we can use an alternative sampler in continuous space state based on Stochastic Gradient Langevin Dynamics (SGLD) [5, 41, 53, 67, 70]:where \(\pi ({\textbf{x}})\) is a prior distribution from which a sample can easily be drawn (e. g., a Gaussian distribution), \(\eta \) is the step size, and \(\epsilon \) is an additive Gaussian noise. This update equation adapts a random initial noise by evaluating the gradients of the modeled probability density and behaves like gradient descent toward lower energy values.

We recall that EBMs estimate the probability density \(q_{\theta }({\textbf{x}})\) over the observation space with a high likelihood for data that are part of \(\mathcal {X}_{\text {train}}\). In contrast, predictive classification models want to estimate the conditional distribution \(q_{\theta }(y\vert {\textbf{x}})\) for data points \({\textbf{x}}\in {\mathbb {R}}^{D}\) and labels \(y\in {\mathbb {R}}\) over a number of distinct classes \(C\). In practice, this distribution is most often estimated using a parametric mapping \(f_{\theta }({\textbf{x}}): {\mathbb {R}}^{D}\mapsto {\mathbb {R}}^{C}\) that assigns real numbers (logits) for all classes to each data point. The logits are subsequently normalized to obtain the categorical distribution aswhere \(\exp (f_{\theta }({\textbf{x}})[y])\) is the \(y^{\text {th}}\) logit.

As shown by Grathwohl et al. [28], both these concepts can be elegantly combined by interpreting the classifier’s logits as unnormalized densities of the joint distribution \(q_{\theta }({\textbf{x}}, y) = \exp (f_{\theta }({\textbf{x}})[y])/Z(\theta )\). In other words, we can use the logits of a discriminative classifier to define an EBM using the relation \(E_{\theta } = -f_{\theta }({\textbf{x}})[y]\).

The marginal distribution over \({\textbf{x}}\) then defines a density model over the data \({\textbf{x}}\) asTraining \(f_{\theta }\) with respect to the joint distribution can be efficiently done using the factorization \(\log q_{\theta }({\textbf{x}},y) = \log q_{\theta }({\textbf{x}}) + \log q_{\theta }(y\vert {\textbf{x}})\) [28, 56]. This allows us to optimize the categorical distribution \(q_{\theta }(y\vert {\textbf{x}})\) using the cross-entropy objective function and to optimize the densities \(q_{\theta }({\textbf{x}})\) with CD. The resulting hybrid model (cf. [49]) with generative and predictive components is known as a Joint Energy-based Model (JEM) [28].

Within SGLD, we can then draw samples from \(q_{\theta }\) in two ways: (1) Computing gradients with respect to \(\log \sum _{y}{\exp (f_{\theta }({\textbf{x}})[y]))}\), which corresponds to unconditional sampling. (2) Computing gradients with respect to \(\log \exp (f_{\theta }({\textbf{x}})[y])\), which allows class-conditional sampling.

One characteristic trait of EBMs is their ability to assign substantially different energy states to ID data and data suffering from semantic or covariate shift, more generally called OOD data. Thus, EBMs implicitly are competitive classifiers to distinguish between ID and OOD data.

Critically, however, optimization with CD does not enforce steady energy states for data sampled from \(p({\textbf{x}})\) and \(q_{\theta }({\textbf{x}})\) as it is unaffected by any additive bias on the energy function’s output. The energy magnitudes fluctuate across the CD iterations and might drift into numerically unstable ranges, which negatively affects the contrastive shape of the energy surface by introducing steep gradients and making it increasingly non-smooth. This process leads to overall worse approximations to the expectation of the energy w. r. t. the model distribution, such that finding an appropriate discrimination threshold for OOD detection becomes very difficult.

An intuitive and simple concept to counteract this behavior is to establish explicit energy levels for \(p({\textbf{x}})\) and \(q_{\theta }({\textbf{x}})\) during CD. Based on the idea of Du et al. [20], we add \(L2\) regularization to constrain the Lipshitz constant, i. e., the in-distribution loss becomes:with \({\textbf{x}}^{+}\sim p({\textbf{x}})\) and \({\textbf{x}}^{-}\sim q_{\theta }({\textbf{x}})\). This regularization term ultimately enforces an energy around \(E_{\theta }({\textbf{x}}^{+})\approx 0\). Because of the remaining approximation bias in \(q_{\theta }({\textbf{x}})\), the energy for \({\textbf{x}}^{-}\) typically ends up being a little bit higher.

In addition, we are often interested in specific scenarios where we know well in advance certain characteristics of OOD data that might interact with our system. This information is particularly valuable when the characteristics relevant for distinguishing between ID and OOD data differ only by nuances. We can assist the contrastive optimization scheme by explicitly introducing known (and task-specific) samples for which we enforce a high energy level.

Let us consider a secondary dataset \(\mathcal {X}_{\text {OOD}}\) with known OOD examples \({\textbf{x}}^{*}\in {\mathbb {R}}^{D}\). For each CD iteration, we can draw a random sample from this dataset and penalize the assigned energy value \(E_{\theta }({\textbf{x}}^{*})\) if it is below a certain margin \(\tau \in {\mathbb {R}}\). We build here on the idea of Liu et al. [46] and implement this penalty term using a squared Hinge loss asThe effect of this energy regularization for OOD data is illustrated in Fig. 2.

A defining concept in CD is to numerically approximate the expectation of the target probability density using an MCMC estimator. To find a good estimate in a computationally efficient manner, we must make sure to focus on regions in the sample space that contribute significantly to the expectation [3]. An intuitive strategy for sampling the model distribution is to explore regions around the mode where we observe high density. This objective agrees well with the concept of maximum likelihood models but is unable to take into account the geometric composition of high-dimensional parametric spaces. Specifically, regions of high probability density lie in a compact neighborhood around the mode but constitute only a small volume. On the other hand, regions of high volume but low density are located toward the tails of the distribution [3]. This effect strengthens and consolidates with increasing dimensionality, such that explorations of the sample space that are based on high-density sampling inevitably ignore important parts of the contributions to the expectation. Note that the strength of this contribution at any point is determined by the product of density \(q_{\theta }({\textbf{x}})\) and volume \(d{\textbf{x}}\), known as the probability mass. Consequently, in order to locate the areas with the most significant contributions to the expectation, we have to carefully consider both the density and the volume. The region where the highest mass concentrates is located between the two extreme points and is generally known as the typical set [3, 12, 50].

This notion of a typical set plays a decisive role when modeling Markov chains. For a well-structured target distribution that is preserved under some Markov transitions, the Markov chain first navigates toward the typical set and then explores it further to reduce the strong initial bias until consolidation. Unfortunately, this process may be disrupted as the Markov transitions are limited in their ability to effectively probe certain types of regions, e. g., high-curvature non-smooth neighborhoods which are very common in practice. For low mixing times, this leads to heavily biased Monte Carlo estimators. With longer mixing times, the Markov chain can sporadically explore these regions, which, however, leads to substantial oscillation in expectation. As a result, running a finite Markov chain in presence of such pathological areas most likely leads to biased and sub-optimal estimators which never reach the true distribution.

It can therefore be advantageous to calibrate the Markov transitions to the geometry of the typical set by querying the target distribution. We propose a lightweight and efficient implementation of this idea by introducing a correction term to the SGLD update equation [Eq. (2)]. This correction harnesses information about the typical set’s geometry at each update step to prioritize Markov states that maintain a certain level of probability mass (Fig. 3).

As a notion for the probability mass at a point, \(q_{\theta }({\textbf{x}}) d{\textbf{x}}\), we use a differential approximation presented by Gratwohl et al. [28] using gradients of the log-density/energy function. As an approximation, the Euclidean norm of these gradients should be small for areas of high mass. Formally, this relation can be written as follows:whereThis correction term can then be used during SGLD update to steer the Markov transitions along a trajectory with sufficient probability mass toward the typical set:\(\gamma \in [0,1]\) is a relative weighting factor that can be tuned to find a balance between high-density and high-mass sampling. In the following, we call this concept mass-corrected sampling (MCS).

Putting it all together, we arrive at the following objective for JEM training:\(\mathcal {L}_{\text {ML}}\) marks the CD policy [Eq. (1)] to optimize the densities \(q_{\theta }({\textbf{x}})\) with the standard biased SGLD sampler according to Eq. (2), or the mass-corrected sampler according to Eq. (9). \(\mathcal {L}_{\text {CLS}}\) with weight \(\alpha \in {\mathbb {R}}\) marks a cross-entropy objective used to optimize the predictive component \(q_{\theta }(y\vert {\textbf{x}})\). \(\mathcal {L}_{\text {ID}}\) with weight \(\beta \in {\mathbb {R}}\) corresponds to the L2 regularization given in Eq. (5), and \(\mathcal {L}_{\text {OOD}}\) with weight \(\lambda \in {\mathbb {R}}\) is the energy margin penalization for OOD data given by Eq. (6). Unless stated differently, we train the models using Eq. (10) with weights \(\alpha =1\), \(\beta =0.1\), \(\lambda =0\), and default SGLD (Eq. (9)) with \(\gamma =1\).

All images were converted from RGB to grayscale and were padded with zeros according to the largest edge in height and width dimensions present in the incunable glyph data. The smaller image dimension was then reshaped to \(56\,\textrm{px}\) while matching the larger dimension according to the original aspect ratio. Lastly, we center-cropped the images to a uniform spatial resolution of \(56 \times 56\,\textrm{px}\), facilitating mini-batch training. Importantly, these steps maintain the relative scale between the image samples.

We use a lightweight Convolutional Neural Network (CNN) for all experiments and ablation studies. The network topology is provided in Appendix Table 5.

We use a maximum epoch size of 120, an Adam optimizer with a default learning rate of 0.0001 with exponential decay every second epoch with a factor of 0.97, and a batch size of 32 across all datasets. We further configure the biased SGLD sampler with \(K=60\) MCMC cycles, a large learning rate \(\eta =20\), and a reduced amount of injected noise with \(\sigma =0.005\), promising faster convergence in scenarios with little cycles \(K\) albeit straining the balance between the sample variance during stochastic gradient descent (early mixing) and the variance during posterior sampling via SLGD (late mixing) [67].

In addition, we use the concept of reservoir sampling/replay buffer [19, 20] to reduce the Markov Chain mixing times by using past samples to initialize SGLD sampling. For each mini-batch, we use 95 % buffer images and 5 % new Gaussian noise samples.

Classification    We evaluate the classification performance using two threshold-independent metrics: (1) we compute the area under the precision-recall curve (AUPRC) using a one-vs-rest approach, and (2) we evaluate the area under the receiver operating characteristic curve (AUROC). We also report the expected calibration error (ECE) [29] to measure the alignment between the predicted softmax probabilities and the classification accuracy.

We start by comparing the classification performance between a discriminative baseline model (CNN) and unconditional and class-conditional JEMs in default configuration (cf. Sect. 3.5). The results are summarized in Table 1.

The discriminative model (CNNs) shows a strong classification performance with an AUPRC of 0.972 and excellent confidence calibration. Despite being additionally tasked with density estimation, the JEM with unconditional sampling even improves beyond this predictive model with a minor increase in AUPRC, albeit at a higher calibration error. As can be observed in Fig. 4, the most common characteristics among the wrongly assigned images include margin crops, lack of centering, label noise (ligatures, rubrications), or partial under-inking which leads to unfavorable similarity to other classes. Moreover, underrepresented classes at the long tail suffer from diffuse predictive probabilities, so that assignments to similar classes are very likely (e. g., ‘k’ \(\rightarrow \) ‘h,’ ‘G’ \(\rightarrow \) ‘E’).

Interestingly, class-conditional sampling (JEM-CC) leads to noticeably worse results with a difference of one percentage point to the unconditional variant (Table 1). By design, class-conditional sampling forces the Markov Chain to concentrate into a distinct and more singular area where the probability mass for the specific target class is high. Given that we start the Markov Chain with a random initial point in the sample space, this region may be distant or may require a trajectory across or out of regions with high contribution to the expectation of visually very similar classes. Although this confounding effect is heavily reduced by using a class-specific sampling reservoir (as we initialize the sampling at nearby points), it could be even further alleviated by increasing the Markov Chain mixing time to compensate for the longer or more complex trajectories.

In this section, we compare the OOD detection performance of the hybrid JEM models with different discriminative inference-time detection methods. In particular, we evaluate different choices of the score function \(s_{\theta }({\textbf{x}})\in {\mathbb {R}}\) for ID and OOD samples and evaluate the separability of the two resulting score distributions (cf. Appendix Table 6).

We consider three OOD detection strategies that make use of the classifiers’ softmax scores. The baseline method by Hendrycks et al. [31] uses the magnitude of the classifier’s highest softmax score, which we refer to as predictive probability. Intuitively, ID images should be assigned a single high class score close to one, whereas OOD samples are expected to yield a rather diffuse probability, ideally approximating a uniform probability over all classes. In ODIN [45], the authors build on this idea and use temperature scaling and input perturbations based on the image gradients to increase calibration of the prediction confidence (cf. [29]), allowing to push apart the predictive probabilities between ID and OOD image samples. While ODIN exploits the gradient domain only implicitly to calculate perturbations, GradNorm by Huang et al. [36] directly derives scores from the gradient space by calculating the vector norm of the gradients backpropagated from the Kullback–Leibler divergence between the softmax output and a uniform distribution. We observed that the scores obtained with this approach were significantly higher for ID data than for OOD data. Moreover, we evaluate score functions based on the free energy score across all class logits [46], as well as on the Mahalanobis distance of a sample to the nearest class-conditional Gaussian distribution using a notion of an induced generative classifier under Gaussian discriminant analysis (GDA) [42]. This strong assumption of the class-conditional distribution in the feature space was pointed out by Sun et al. [63] as a potential weakness. They proposed to use a non-parametric estimate using k-th nearest neighbor (KNN) distances between the trained ID embeddings and the embedding of the input sample. For the hybrid models, we use again predictive probability, which performed better than energy-based scores based on probability density and mass during an initial analysis on the OOD datasets’ validation splits (cf. Appendix Table 7).

The results in Table 2 reveal that most OOD detection methods reliably separate the clutter images from ID data. Surprisingly, applying GradNorm leads to a much worse separation of the two distributions, suggesting that the differences in gradient magnitude are not sufficiently discriminative. We hypothesize that the inter-class differences in the glyph are smaller than those in the ID datasets considered in the original work, which might lead to less informative gradient norms. The KNN method yields the best results by a large margin. In comparison, all discriminative methods yield noticeably worse detection rates for the OOD ligature data. Interestingly, an additional calibration of the prediction confidence does not aid OOD detection. For example, ODIN utilizes perturbations of the input to increase the maximum softmax score of the ID samples. As we already observe exceptionally high scores for the ID data at well-calibrated confidence (cf. Table 1), the effect is seemingly negligible. It should also be noted that neural networks in many scenarios tend to produce overconfident predictions [30, 32, 43, 47, 52]. These scores can also be disproportionately high for OOD data, even if the data is far away from the softmax decision boundary [42], which curtails the effectiveness of ODIN further.

Using the Mahalanobis distance score significantly improves the ligature detection rate in terms of AUPRC and AUROC scores compared to the predictive probability baseline. At the same time, the FPR at a fixed threshold of 95 % TPR is shown to be worse than the predictive probability baseline, highlighting the importance of adequate threshold selection. Because the class-conditional distributions relevant for calculating the distance-based scores are derived using features from different receptive fields, we argue that both local details and global context are important to capture the intricate differences between stand-alone and linked allographs. For example, ligatures starting with the character ‘f’ might connect to the adjacent character by a slight extension of the ascender (e. g., ‘l’), the crossbar (e. g., ‘o’), or might only show an infiltration into the baseline region without a continuous ink trace.

Similar to the observations on the clutter data, the KNN-based OOD detection yields very competitive AUROC and FPR95 scores that substantially improve over the predictive probability baseline. However, while this tells us that KNN produces very few false positives (ID samples assigned to the OOD distribution) given a high TPR, the low AUPRC indicates that we still obtain a relatively large absolute count of false positives. Furthermore, KNN explicitly uses only feature embeddings from the penultimate layer to the nearest neighbor distances. For that reason and based on the observation of the Mahalanobis distance score, it would be interesting to investigate whether a weighted addition of earlier network layers with a stronger local context affects detection quality.

The unconditional JEM without any further modifications matches the OOD detection rate of the best-performing discriminative models for ligatures and clutter images. In comparison, actuating the trend in classification results, the class-conditional JEM achieves worse results than the unconditional variant, which is particularly evident in the poorer detection of ligatures. The four percentage point difference in AUPRC performance suggests that it is useful to capture overarching features across all classes, such as often occurring linking patterns between adjacent characters.

In this section, we introduce energy margins between ID and OOD data with an additional Hinge loss term (cf. Sect. 3.3) and evaluate its influence on classification and OOD detection.

We begin by exploring the hyper-parameter space for different combinations of the energy margin \(\tau \in \{3,6,9\}\) and the multiplicative weight \(\lambda \in \{0.01, 0.03, 0.1\}\). Because we are interested in high classification accuracy and high OOD detection performance at the same time, we select the hyper-parameters according to the highest harmonic mean of classification AUPRC and OOD AUPRC for clutter and ligatures on the validation sets (Appendix Fig. 8). The best-performing configurations were found to be \((\tau =3;\,\lambda =0.03)\) for JEM and \((\tau =3;\,\lambda =0.01)\) for JEM-CC. We observed that the class-conditional variant suffers from a diverging CD training loss with eventual breakout for almost all runs with higher margins or weights. While this phenomenon affects the classification abilities only marginally, it impairs the OOD detection on both OOD datasets and leads to unstable training objectives.

We now apply the models with a tuned energy margin on the test set (Table 3). The explicit separation of clutter images yields only minor improvements to the OOD detection scores as these images are already assigned a sufficiently large energy value by default. This can be explained by a large difference in observed characteristics between members of the target classes and the considered types of clutter. Conversely, enforcing a margin between ID data and ligature images shows over 2 percentage points improvement in OOD detection AUPRC and reduces the FPR95 by 17.2%. This detection performance is also notably better than that of the KNN method, especially in the absolute number of false positive assignments to the OOD distribution. Here, providing the network with hard OOD samples to shape the energy surface in between ID and OOD data distributions helps to pinpoint intricate visual differences despite the strong semantic similarity. For both unconditional and class-conditional modeling, the classification performance slightly drops when certain energy states are enforced.

A particularly interesting observation can be made for the models that are trained on the combined OOD data pool. Although the OOD detection rates can be improved for both clutter and ligatures, the magnitude of the improvement falls short of the specialized training variants. This effect suggests that different classes of OOD data should not be assigned to the same region of the energy spectrum but rather be treated separately. In this context, future work could investigate into treating different OOD data as additional classes during training of the predictive component.

We continue by introducing a notion of probability mass to the SGLD update equation (i. e., mass-corrected sampling) and analyze its effects on training stability and sampling quality (cf. Sect. 3.4). The results are summarized in Table 3.

In general, MCS shows no effect when it is used to train unconditional models. For class-conditional sampling, however, MCS leads to a significant improvement in discriminating between ID images and ligatures with 4.7 percentage points in AUPRC. This trend is reinforced when MCS is used together with an explicit ID/OOD energy margin, albeit at the cost of a noticeable decrease in classification accuracy. From a conceptional perspective, training with a class-conditional sampler requires a comparably sharper separation of class-specific densities in sample space. This effect makes it difficult to fulfill the contrastive update scheme, which builds on the idea of increasing the likelihood of a sample given a specific class while at the same time pulling down the likelihood of samples from the model distribution and known OOD samples. We argue that in such a scenario, MCS helps to stabilize the SGLD update trajectory in more pathological regions by favoring Markov transitions that do not move away too far from the typical set [3].

The visualization of the CD training loss in Fig. 5 shows that this stabilizing effect prevents loss breakouts and the subsequent divergence of the optimization process. This is particularly true for more aggressive hyper-parameter selections with large energy margins \(\tau \) and weighting factors \(\lambda \).

This section examines the hybrid model’s behavior when exposed to images affected by different degradations frequently observed on historical printed documents. Specifically, we are interested in the impact of these effects on classification and OOD detection performance. die häufige Probleme bei historischen Drucken abbilden. We consider six different image augmentations which are applied and evaluated in an isolated manner.

The results for each augmentation type are visualized in Fig. 6. With increasing additive Gaussian noise up to a standard deviation of 0.75 px, the classification performance and OOD detection rate remain almost constant. For more aggressive noise configurations, the augmented images increasingly resemble the initial Markov states in SGLD sampling which correspond to very high energies. Furthermore, decreasing the image resolution until the point of losing most of the visual characteristics of the original letter (at 25 % of original resolution) has no considerable effect on the performance metrics. Given that historical documents were produced to inconsistent standards, show strongly varying traces of age or use, and were scanned using a wide array of scanners, cameras, and recording protocols over the past decades, this robustness of the model to different resolutions is a powerful property.

Exposing the network to images with fewer amounts of applied ink (erosion) quite quickly leads to a noticeable drop in classification and OOD detection performance. A particularly interesting observation can be made for the input density which barely drops with different sizes of the structuring element, indicating that the characteristics of under-inking have significant contributions to the expectation. This is in contrast to the quickly decreasing maximum softmax score. Consequently, it can be concluded that also for this scenario an OOD score function based on the predictive part of the hybrid model is the better choice. The decrease in performance is less pronounced for broadening of the characters (dilation) where the drop only occurs after reaching unrealistic levels of ink coverage. Compared to the erosion results, the value by which the energy scores increase is significantly higher as the size of the structuring element increases. It can be concluded that over-inking plays a minor role in the present data set. At the same time, the classification performance remains stable even at moderate dilation rates.

Figure 6 further reveals that even for very aggressive JPEG compression rates both OOD detection scores and the performance metrics hold up very nicely. This is an important property since a considerable amount of scanned material is made available to the public in compressed JPEG format. Also, minor skewing of the scanning page with rotation angles of \(\pm 10^\circ \) still yields sufficient performance scores.

In this section, we exploit the generative capabilities of our model for class-conditional sampling. Using the notion of probability mass (Eq. (9)), we seek to visualize image characteristics that play a significant role in optimizing the model distribution \(q_{\theta }({\textbf{x}})\). The generated class prototypes for different lower- and uppercase characters are shown in Fig. 7.

In general, the considered classification task works on a rather standardized type design where variance is mostly contained in smaller local details. This variance includes different amounts of ink along the centerline of the character, age-related degradations (e. g., smears, smudges), deviations between different metal types due to wear, and various types of allographs and their defining typographic properties (e. g., contraction signs like overline or curved abbreviation stroke). These characteristics are well captured across several generated images. Moreover, for characters like ‘t’ with curved abbreviation stroke (Latin abbreviation for -tur or less often -er), we observe reoccurring visual features like a small part of the adjacent letters, indicating that the model successfully captures semantics in the margins of the glyph images.

Currently, method evaluation is done on a single digital reproduction only. This might seem a manageable problem at first, as incunables comprise only a relatively small percentage of the total body of letterpress prints with moderate variance in fonts and typographic styles. However, their digital analysis must deal with a tremendous variety of scanner types, acquisition protocols, and the state of conservation which differ between libraries, editions, copies and individual reproductions. It is, therefore, indispensable for a widely usable application to either train very specialized models or to use a single model operating as generally as possible. Furthermore, the proposed fine-tuning technique at its core implements a data-driven pruning algorithm to extract only the most probable images and discard implausible ones. If the downstream task requires (near-) full coverage, this rather aggressive approach might not be appropriate. In this case, optimization efforts should concentrate on building more robust OCR models, albeit requiring additional laborious transcription work. Lastly, during method development, we observed that larger CNNs like the Wide Residual Network [72] or the Vision Transformer [15] typically converge to odd glyph shapes. Similar behavior was already described by Nijkamp et al. [54] and Du et al. [19], confirming the generally known stability problems of EBMs [20]. With this in mind, we want to point out to the reader that there are also alternative options for training EBMs that could help circumvent these issues (cf. [27]).

Apart from exploring the method’s applicability to new types and compositions of data, we see strong merit in investigating different and possibly less ‘invasive’ techniques to integrate prior knowledge of OOD data. It would be particularly interesting to encourage the model to optimize toward a joint distribution of ID and OOD data without externally enforcing specific energy levels. For the application to early printed documents with often very localized variations in shape, this direction might prove beneficial. Furthermore, we want to investigate compositional patterns of specialized EBMs using different glyph concepts. As shown by Du et al. [16], we can use concept negation to learn a model distribution over data that does not include certain concepts, which might be viable for the goal of OOD detection.