Generating Machine-Style Handwriting: A Diffusion Based Latent Generation with VAE Decoding

In this paper, we introduce the Style-Calligraphy model, an innovative architecture designed to generate high-fidelity images of text in specified machine styles, conditioned on a given text input. Our approach leverages the strengths of Variational Autoencoders (VAEs) and Latent Diffusion Models (LDMs) to address the challenges of latent space representation and efficient image generation. The VAE encoder-decoder framework is employed to learn structured latent spaces, mitigating the limitations of traditional autoencoders by incorporating Kullback-Leibler divergence alongside image reconstruction loss. This ensures a continuous and feasible latent space for sampling. The LDM is trained as a denoiser with text-based conditioning, utilizing a Markov chain to model the noise addition process and employing cross-attention mechanisms to enhance spatial character relationships. We introduce a novel sliding cross-attention technique using duplets and triplets to capture intricate dependencies between characters, significantly improving the model’s performance. Furthermore, we propose a stand-alone image decoder to address noise sensitivity, trained on both clean and noisy latent representations, resulting in a substantial increase in image quality. A key innovation of our work is the repurposing of a single LDM across multiple machine styles, drastically reducing training costs by isolating style-specific training to the image decoder. Our comprehensive training pipeline, optimized for efficiency, demonstrates the model’s capability to generate accurate and stylistically coherent text images, achieving a 99.5% success rate in high-quality sample generation on seen data.