A comparison of methods for sketch-based 3D shape retrieval

Highlights

• Build a small scale and a large scale sketch-based 3D model retrieval benchmark.

• Evaluate 15 best sketch-based 3D model retrieval algorithms on the two benchmarks.

• Solicit and identify the state-of-the-art methods and promising related techniques.

• Incisive analysis on diverse methods w.r.t scalability and efficiency performance.

• The benchmarks and evaluation tools provide good reference to the related community.

Abstract

Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. To foster this research area, two Shape Retrieval Contest (SHREC) tracks on this topic have been organized by us in 2012 and 2013 based on a small-scale and large-scale benchmarks, respectively. Six and five (nine in total) distinct sketch-based 3D shape retrieval methods have competed each other in these two contests, respectively. To measure and compare the performance of the top participating and other existing promising sketch-based 3D shape retrieval methods and solicit the state-of-the-art approaches, we perform a more comprehensive comparison of fifteen best (four top participating algorithms and eleven additional state-of-the-art methods) retrieval methods by completing the evaluation of each method on both benchmarks. The benchmarks, results, and evaluation tools for the two tracks are publicly available on our websites [1], [2].

Introduction

Sketch-based 3D model retrieval is focusing on retrieving relevant 3D models using sketch(es) as input. This intuitive and convenient scheme is easy for users to learn and use to search for 3D models. It is also popular and important for related applications such as sketch-based modeling and recognition, as well as 3D animation production via 3D reconstruction of a scene of 2D storyboard [3].

However, most existing 3D model retrieval algorithms target the Query-by-Model framework, that is, using existing 3D models as queries. In the areas of content-based 2D image retrieval and image synthesis, sketch-based methods have been addressed for some time now. In 3D model retrieval, on the other hand, less work has to date considered the Query-by-Sketch framework. In fact, it is a non-trivial task to perform sketch-based 3D model retrieval and also more difficult compared with the Query-by-Model case. This is because there exists a semantic gap between the sketches humans draw and the 3D models in the database, implying that the structure of the query and target objects differ. Specifically, target objects are typically given as precisely modeled objects, while the query sketch may differ drastically in level of detail, abstraction, and precision. In addition, until now there is no comprehensive evaluation or comparison for the large number of available sketch-based retrieval algorithms. Considering this, we organized the Shape Retrieval Contest (SHREC) 2012 track on Sketch-Based 3D Shape Retrieval [1], [4], held in conjunction with the fifth Eurographics Workshop on 3D Object Retrieval, to foster this challenging research area by providing a common small-scale sketch-based retrieval benchmark and soliciting retrieval results from current state-of-the-art retrieval methods for comparison. We also provided corresponding evaluation code for computing a set of performance metrics similar to those typically used to evaluate Query-by-Model techniques. The objective of this track was to evaluate the performance of different sketch-based 3D model retrieval algorithms using both hand-drawn and standard line drawings sketch queries on a watertight 3D model dataset. Every participant performed the queries and sent us their retrieval results. We then did the performance assessment.

A satisfactory success has been achieved in the SHREC’12 sketch track [4]. However, the contest has limitations in terms of its evaluation of different sketch-based retrieval algorithms based on a rather small benchmark and a comparison of a limited number of methods. Eitz et al. [5] provided us the largest sketch-based 3D shape retrieval benchmark until 2012, based on the Princeton Shape Benchmark (PSB) [6] with one user sketch for each PSB model. However, until now no comparative evaluation has been done on a very large-scale sketch-based 3D shape retrieval benchmark. Considering this and encouraged by the successful sketch-based 3D model retrieval track in SHREC’12 [4], in 2013 we organized another track [2], [7] with a similar topic in SHREC’13 to further foster this challenging research area by building a very large-scale benchmark and soliciting retrieval results from current state-of-the-art retrieval methods for comparison. Similarly, we also provided corresponding evaluation code for computing the same set of performance metrics as the SHREC’12 sketch track. For this track, the objective was evaluating the performance of different sketch-based 3D model retrieval algorithms using a large-scale hand-drawn sketch query dataset for querying from a generic 3D model dataset.

After finishing the above two SHREC contests, we have found that the participating methods for the two contests are not completely the same, thus a conclusion of the current state-of-the-art algorithm is still unavailable. In addition, to provide a more complete reference for the researchers in this research direction, it is necessary to perform a more incisive analysis on different participating methods w.r.t their scalability and efficiency performance, as well as the two benchmarks used in the two contest tracks. Motivated by the above two findings, we decided to perform a follow-up study by completing a more comprehensive evaluation of currently available top sketch-based retrieval algorithms on the two benchmarks such as to perform a more comprehensive comparison on them and solicit the state-of-the-art approaches. Thus, we sent invitations to the participants as well as the authors of recently published related papers (according to our knowledge) to ask them to contribute to the new comprehensive evaluation. Totally, 6 groups accepted our invitations and agreed to submit their results on schedule. Finally, 15 best-performing methods (4 top participating algorithms and 11 additional state-of-the-art approaches; totally 17 runs) from 4 groups successfully submitted their results, including running results (e.g. retrieval lists and timing information) and method description, which are also available on the SHREC’12 and SHREC’13 sketch track website [1], [2]. After that, we performed a comparative evaluation on them.

In this paper, we first review the related work (w.r.t. techniques and benchmarks, respectively) in Section 2. Then, in Section 3 we introduce the two benchmarks (one small-scale and one large-scale) used in the two contest tracks. Section 4 gives a brief introduction of the contributors of the paper. A short and concise description for each contributed method is presented in Section 5. Section 6 describes the evaluation results of the 15 sketch-based 3D retrieval algorithms on the SHREC’12 small-scale benchmark and SHREC’13 large-scale benchmark, respectively. Section 7 further comments on the benchmarks and analyzes the contributed algorithms w.r.t the performance they achieved. Section 8 concludes the paper and further lists several future research directions.