Evaluating reward and risk for vertical selection

The aggregation of search results from heterogeneous verticals (news, videos, blogs, etc) has become an important consideration in search. When aiming to select suitable verticals, from which items are selected to be shown along with the standard "ten blue links", there exists the potential to both help (selecting relevant verticals) and harm (selecting irrelevant verticals) the existing result set.

In this paper, we present an approach that considers both reward and risk within the task of vertical selection (VS). We propose a novel risk-aware VS evaluation metric that incorporates users' risk-levels and users' individual preference of verticals. Using the proposed metric, we present a detailed analysis of both reward and risk of current resource selection approaches within a multi-label classification framework. The results bring insights into the effectiveness and robustness of current vertical selection approaches.