Inv3D: a high-resolution 3D invoice dataset for template-guided single-image document unwarping

Numerous business workflows in enterprises involve printed forms, such as invoices, bills, or receipts. The receiving party has to manually digitalize the document in order to persistently access, search, or store the provided data, causing significant personnel costs. Here, existing solutions make use of scanners to create flatbed digital copies of the paper document and apply optical character recognition (OCR) to automatically extract information. This, however, creates additional costs and reduces the flexibility of the given solution.

In order to overcome the hardware restriction, current state-of-the-art approaches attempt to analyze document images taken with smartphones. Prominent examples are DewarpNet [6] and GeoTr [9] which learn to dewarp images using supervised learning, having the available dewarping meshes as ground truth. While these approaches generate promising results, they still are not satisfactorily robust when dealing with environmental factors, such as light incidence, shadows, occlusions, crumpled or folded paper, and perspective transformations.

One potential remedy is the use of additional structured information in the form of templates, which confine the general structure of the documents in order to improve unwarping. While it might be tedious to define initial templates, the added value is significant due to the considerable increase in dewarping precision and robustness.

In this paper, we follow exactly this path and propose a novel labeled invoice dataset with additional structural information to assist image dewarping. More specifically, we present Inv3D, a large, high-resolution invoice dataset comprising both synthetic data generated from carefully designed templates and challenging real-world data. Inv3D consists of 25,000 samples, each composed of four flatbed invoice image layers, two ground-truth annotations, the 3D warped document, nine supervision signal maps, and the backward transformation map (see Fig. 1). We then propose a novel supervised dewarping approach, referred to as GeoTrTemplate, which exploits the novel structured information by extending the recent GeoTr [9] algorithm. We encode both the warped image and our template image using a convolutional neural network and combine the feature representations. The subsequent transformer encoder–decoder module learns the attention between all pairs of features which enables the model to combine the warped image with our structural template information. For our extensive evaluation, we trained DewarpNet [6] without refinement network and GeoTr [9] in a unified framework and evaluate both approaches on the Doc3D dataset, as well as Inv3D, making use of established metrics such as MS-SSIM, LD, ED, and CER. In addition, we introduce the newer perceptual metric LPIPS [50] as a benchmark metric for document dewarping. To the best of our knowledge, this is the first dataset and approach to make use of structured template information for available invoices to foster research on robust image dewarping systems.

Our contributions are threefold:The paper is structured as follows: First, we introduce the related work and then we explain the creation process of Inv3D and present our own approach before reporting our evaluation. We conclude with Sect. 6.

For the extraction of textual information from images, several factors strongly influence the performance of off-the-shelf solutions such as Tesseract [38]. Images from flatbed scanners have constant illumination, little noise, and no deformations. Since these factors are important for the accuracy of OCR, there has been research to reconstruct those conditions from images of documents that were captured in the wild.

Datasets. There are several datasets available that focus on document rectification, which are summarized in Table 1. An early real dataset comprising 102 images of bend documents for evaluation was presented for a page dewarping contest [35]. A big synthetic dataset focusing on bends only by employing the cylindrical surface model from Cao et al. [2] was presented by Garai et al. [12]. One step forward toward higher complexity of deformations and thus toward higher applicability for real-world applications was presented by Ma et al. [29]. They created a large synthetic dataset comprising bends and folds. Deformations were generated in 2D and thus generalize poorly on 3D structures. This dataset was extended by RectiNet [1]. The current state-of-the-art dataset is Doc3D [6]. It is a large document unwarping dataset comprising captured 3D meshes and a collection of rendered document images. It adds more realism by including crumpled documents. CREASE [31] follows the pipeline of Das et al. [6] to render a high-resolution dataset with additional supervisory signals, which, however, is not publicly available. Inv3D represents the first publicly accessible dataset with high resolution and complex 3D structures. Similar to CREASE, we provide additional 3D annotations such as per-pixel angle, curvature, and text maps. In addition to that, Inv3D is the first template-based dataset that enables research on structured documents in the warped domain with the support of flatbed template information while comprising the same challenging aspects as Doc3D. The exploitation of known flatbed templates at inference time might ease the problem of image dewarping and thus improve the potential for real-world applications of this technology.

Approaches. The correction of illumination and noise in document images has been considered in [7, 9, 21, 33, 41]. Also, there is work on improving text extraction from noisy images [16]. The task of document rectification, i.e., generating a flatbed version of the captured document, is at the core of our work and thus reviewed in more detail.

While there has been work using 3D sensors for document unwarping [4, 23, 40, 47], we are interested in recovering a flatbed version of documents from a single image only. Features in 2D have played an important role for this task. Popular examples include the usage of baselines of horizontal text [14, 17, 18, 26, 36] and vertical stroke boundaries [27, 28]. These feature-driven approaches are frequently combined with the assumption of a cylindrical surface model [2] or a developable surface [22]. Tian and Narasimhan [39] relaxed the assumption on the surface model by allowing to reconstruct a broader class of 3D shapes. A CNN-based approach for texts in English and Bangla was presented by Garai et al. [12].

One drawback of the aforementioned works is that they focus on bend surfaces. Folds and especially crumples, which make the geometry of the surface significantly more complex, are neglected despite their common occurrence in document images. The first work using deep learning for distortion estimation, and not only feature extraction, from a single captured image is DocUNet [29]. The usage of this deep learning pipeline significantly sped up the unwarping process compared to previous methods; however, only 2D deformations were used during training. RectiNet [1] uses a gated and bifurcated stacked U-Net module and slightly improves upon the original DocUNet model. DewarpNet [6] improves upon DocUnet by incorporating 3D information. A two-stage system with two sub-networks, one for unwarping and one for texture mapping, is presented. Xie et al. [43] introduced a deep learning-based method that uses displacement flow estimation. Here, Doc3D is used as a dataset. CREASE [31] introduces a context-aware end-to-end dewarping pipeline. They acknowledge the importance of predicting the orientation of the text and add a text angle prediction as previously seen in Scene Text Recognition tasks. Feng et al. [9] introduce a transformer-based model to capture the global context of the document via self-attention. Xie et al. [44] propose a model which learns sparse control points that they use to interpolate the backward mapping. Das et al. [8] decouple local and global unwarping by dividing the image into patches prior to the rectification process and stitching them together afterward. DocScanner [10] and Marior [49] tackle the unwarping problem iteratively using a single rectification estimate in multiple steps. Xue et al. [46] unwarps the documents in two steps using the coarse and the refinement transformer. In contrast to previous work, the authors focus on high-frequency signals for training. Jiang et al. [15] formulate the task as a constrained optimization problem. They segment background/foreground, detect text lines, and use these constraints to search for an unwarping map. The PaperEdge system by Ma et al. [30] pre-unwarps the documents based on the outline of the paper sheet and learns subsequently a texture-based deformation to get the final result. Lastly, [11] propose DocGeoNet, a geometrically constrained representation of the document. The approach learns an intermediate 3D representation of a given document before inferring the backward map. While the state-of-the-art shows promising results, it is still a very active area of research to increase both the quality and reliability of current approaches. We chose to base our approach on the geometric unwarping model DocTr [9] which shows highly competitive unwarping performance. DocTr is especially a good choice since its integrated attention mechanism allows the model to meaningfully integrate the additional structural information in the form of templates.

Since, to the best of our knowledge, there is no large-scale, high-resolution dataset for document dewarping publicly available, which includes visual templates, we fill this gap by creating a novel dataset called Inv3D. The dataset creation pipeline consists of four stages: resource preparation, invoice rendering, invoice warping, and finally the auxiliary map generation. Note that our dataset Inv3D focuses on a single type of documents only, namely invoices. Since we provide our generation pipeline, one can easily create their own dataset using other types of documents. We conclude this section by introducing a new real-world evaluation dataset called Inv3DReal.

In this section, we present our novel approach for image dewarping by leveraging structural templates at training and inference time. We extend the transformer-based state-of-the art model GeoTr introduced by Feng et al. [9] to incorporate the a-priori known structural information represented through the invoice templates. In the following, we refer to our new model as GeoTrTemplate and its extension as GeoTrTemplateLarge. See Fig. 5 for a schematic. Our model receives the warped image \({{\,\mathrm{\textbf{W}}\,}}\in \mathbb {R}^{h_0 \times w_0 \times 3}\) and the template image \({{\,\mathrm{\textbf{T}}\,}}\in \mathbb {R}^{h_1 \times w_1 \times 3}\). Both inputs are scaled to a fixed resolution of \(288 \times 288\) for GeoTrTemplate and \(600 \times 600\) for GeoTrTemplateLarge before applying a geometric head H individually to each image. The head H creates deep image representations with \(36 \times 36\) positional features in a 128-dimenstional space. For GeoTrTemplate, we employed the geometric head proposed by Feng et al. [9]. We define the geometric head for our large model as a slice of the EfficientNet B7 noisy student model [45], namely the first four blocks followed by a convolutional layer. The features of the warped image \(H({{\,\mathrm{\textbf{W}}\,}})\) and the template \(H({{\,\mathrm{\textbf{T}}\,}})\) are concatenated, forming a combined input representation \(R \in \mathbb {R}^{36 \times 36 \times 256}\). We then applied the transformer encoder and decoder from Feng et al. [9] and their geometric tail module to upsample the resulting backward map. For details regarding the employed modules, see the original paper. The output is a backward map \({{\,\mathrm{\textbf{B}}\,}}\in [0, 1]^{288 \times 288 \times 2}\). Our loss function is defined as the L1-norm between the output backward map \({{\,\mathrm{\textbf{B}}\,}}\) and the true backward map \({{\,\mathrm{\varvec{\hat{B}}}\,}}\).

In this work, we presented Inv3D, a novel high-resolution invoice dataset with both synthetic and real-world data with rich label information. Apart from the rectification mesh for dewarping each individual invoice, we add corresponding template information which can be utilized during training for improving generalizability. Inv3D comprises 25,000 samples based on 100 templates and heterogeneous environmental factors, such as challenging lighting conditions and a multiplicity of document deformations. In addition, we introduced GeoTrTemplate and GeoTrTemplateLarge, two novel models which leverage a-priori available structural information for the task of document image dewarping. We conducted a detailed evaluation study to compare our new models with the state-of-the-art approaches DewarpNet [6] and GeoTr [9]. Our empirical analysis showed that both outperform the baseline methods significantly; in particular, the GeoTrTemplateLarge model improves the local distortion of GeoTr by 26.1 %. Nevertheless, the absolute values for text detection show that further research is needed in order to solve this task robustly and consistently.

In future work, we plan to investigate better approaches to exploit template information, as available in Inv3D. The iterative refinement approach proposed by Feng et al. [10] for DocScanner might be used to create a correspondence map of interest points in the warped image and the template. We believe that the availability of templates at inference time can play an important role in document dewarping through the visual cues they provide. Another interesting direction is the calculation of unwarping confidence scores based on the matching of the unwarped image and the template. The confidence scores could become important for real-world applications to avoid the acquisition of erroneous information.