Hierarchical Image Inpainting by a Deep Context Encoder Exploiting Structural Similarity and Saliency Criteria

The purpose of this paper is to present a context learning algorithm for inpainting missing regions using visual features. This encoder learns physical structure and semantic information from the image and this representation differentiates it from simple auto encoders. Such properties are crucial for tasks like image in-painting, classification and detection. Training was performed by patch-wise reconstruction loss using Structural Similarity (SSIM) jointly with an adversarial loss. The reconstruction loss is also augmented using spatially varying saliency maps that increase the error penalty on distinctive regions and thus promote image sharpness. Furthermore, in order to improve image continuity on the boundary of the missing region, distance functions with increasing importance towards the center of the inpainting region are also used either independently or in conjunction with the saliency maps. We also show that our choice of reconstruction loss outperforms conventional criteria such as the L2 norm. This means giving more weight to pixels closer to the border of the missing image parts and also giving more important to salience parts of the image to guide the reconstruction, thus producing more realistic images.