Im2Sketch: Sketch generation by unconflicted perceptual grouping

Abstract

Effectively solving the problem of sketch generation, which aims to produce human-drawing-like sketches from real photographs, opens the door for many vision applications such as sketch-based image retrieval and non-photorealistic rendering. In this paper, we approach automatic sketch generation from a human visual perception perspective. Instead of gathering insights from photographs, for the first time, we extract information from a large pool of human sketches. In particular, we study how multiple Gestalt rules can be encapsulated into a unified perceptual grouping framework for sketch generation. We further show that by solving the problem of Gestalt confliction, i.e., encoding the relative importance of each rule, more similar to human-made sketches can be generated. For that, we release a manually labeled sketch dataset of 96 object categories and 7680 sketches. A novel evaluation framework is proposed to quantify human likeness of machine-generated sketches by examining how well they can be classified using models trained from human data. Finally, we demonstrate the superiority of our sketches under the practical application of sketch-based image retrieval.

Introduction

There exists plenty of prior work on sketch in computer vision, from sketch recognition [1], [2] to sketch-based image retrieval (SBIR) [3], [4]. Recently, sketch research has gained much momentum due to the proliferation of touch sensitive devices. Nonetheless, how to automatically produce sketches using machines as humans do is still an open problem [1], [5], [6]. Solving this problem importantly opens the door for many vision applications, especially for SBIR since better sketch conversion essentially closes the domain gap between query sketches and gallery photographs.

This paper sets out to tackle the problem of sketch generation via learning from established theories found in human visual cognition studies. Human visual system is very powerful so that we can easily find sense from chaos. In Neuroscience, one of the most critical problems is understanding how human brains perceive visual objects. In particular, perceptual grouping, a concept introduced by the Gestalt school of psychologists [7], advocates that human perceives certain elements of the visual world as going together more strongly than the others. Max Wertheimer, a pioneer in the Gestalt school, pointed out the significance of perceptual grouping and further listed a series of rules, such as proximity, similarity and continuation [8]. His work has consequently triggered plenty of research specifically aimed at understanding human visual systems [9], [10].

We treat sketch generation as a perceptual grouping and filtering task. Essentially, the underlying hypothesis is that perceptual grouping is able to find sense from chaos therefore leaving only signals corresponding to sketches of human resemblance. More specifically, our choice of utilizing an elementary grouping process for sketch generation is justified as follows: (i) sketches exhibit a lack of visual cues (black and white drawing versus textured image regions), making conventional vision algorithms sub-optimal, (ii) sketch despite abstract (e.g., a stickman human figure) is very iconic – it is often structural variations of strokes that capture object appearance, last but not least, (iii) sketches are the simplest form of depictions of human visual impressions that can be rendered by hand, therefore act as ideal basis for applying/testing theories found in human visual cognition [3].

Traditionally, applying perceptual grouping in vision applications incurs two critical design considerations: (i) how to combine multiple Gestalt rules into a single globally optimized framework, (ii) how to encode the relative importance of each rule. For the former problem, although unary Gestalt principle has been proven to be useful for contour grouping when used alone [11], [12], [13], very few work [14] attempt to investigate how they can be exploited jointly in a single framework. The latter problem, often referred as Gestalt confliction, remains unaddressed to date [7], [15]. Despite being the subject of investigation in the fields of psychology, little is known about how Gestalt confliction work in human vision systems [16], thus shedding little light on how to design a computer vision system.

In this paper, we first propose a unified grouping framework that is able to work with multiple Gestalt principles simultaneously. We then show how Gestalt confliction can be accounted for by learning from a dataset of pre-segmented human sketches. In particular, a multi-label graph-cuts [17], [18], [19] perceptual grouping framework is developed to group stroke segments while utilizing the learned importance of different Gestalt principles. It follows that, upon generating sketch from photograph, the same learned perceptual grouping framework can be used to form groups of image boundary segments, which are further filtered to produce human-drawing-like sketches. More specifically, a learning to rank strategy based on RankSVM [20] is proposed to learn the relative importance between two Gestalt principles, namely proximity and continuity. We learn from a subset of a large scale human-drawn sketch dataset [1], where each sketch is pre-segmented into semantic groups. The entire process of the proposed sketch generation framework is shown in Fig. 1.

To evaluate the quality of automatically generated sketches, we present a novel approach that recognizes them by sketch classifiers trained from human data. Prior works [5], [21] evaluate sketching performance by comparing computer generated sketches with tracings produced by humans. This evaluation strategy importantly does not account for likeness to human-made sketches because (i) sketches are abstract depictions that are fundamentally different from tracing of image boundaries, (ii) humans often sketch without reference to real photographs of objects, (iii) sketches exhibit much more intra-class variability, due to different levels of drawing skills and individual visual impressions. By measuring how well a human sketch classifier [1] recognizes machine-generated sketches, we essentially examine how closely they resemble human-made ones. Our results show that the sketches generated using our method outperform a number of state-of-the-arts alternatives.

To further demonstrate the quality of our sketches, we demonstrate its effectiveness for SBIR. Experimental results confirm a positive performance boost on the largest SBIR dataset to date [4], when compared with state-of-the-art alternatives. It importantly shows that our sketch generation algorithm is able to bridge the domain gap between sketches and natural images. As can be seen from Fig. 2, the proposed sketch converter yields cleaner sketch that matches better to the query when encoded using common descriptor such as Histogram of Oriented Gradients (HOG).

The contributions of this paper can be summarized as follows:

(1) We apply perceptual grouping as means for automated sketch generation and propose a learning to rank strategy to learn the relative importance among two Gestalt principles.

(2) A novel evaluation strategy is devised to quantitatively evaluate human likeness of sketches.

(3) We demonstrate the effectiveness of sketch generation in SBIR and show a performance boost when compared with state-of-the-art alternatives.

(4) A new dataset containing 96 object categories and 7680 sketches is released, where each sketch is segmented into semantic parts by human.