Fusing Keyword Search and Visual Exploration for Untagged Videos

Video collections often cannot be searched by keywords because most videos are poorly annotated. We present a system that allows to search untagged videos by sketches, example images and keywords. Having analyzed the most frequent search terms and the corresponding images from the Pixabay stock photo agency we derived visual features that allow to search for 20000 keywords. For each keyword we use several image features to be able to cope with large visual and conceptual variations. As the intention of a user searching for an image is unknown, we retrieve thousands of result images (video scenes), which are shown as a visually sorted hierarchical image map. The user can easily find images of interest by dragging and zooming. The visual arrangement of the images is performed with an improved version of a self-sorting map, which allows organizing thousands of images in fractions of a second. If an image similar to the search query has been found, further zooming will show more related images, retrieved from a precomputed image graph. The new approach helps to find untagged images very quickly in an exploratory, incremental way.