file.pdf (483.35 kB)

A Methodology for Evaluating Aggregated Search Results

Download (483.35 kB)
journal contribution
posted on 01.04.2011, 00:00 by Jaime Arguello, Fernando Diaz, Jamie Callan, Ben Carterette

Aggregated search is the task of incorporating results from different specialized search services, or verticals, into Web search results. While most prior work focuses on deciding which verticals to present, the task of deciding where in the Web results to embed the vertical results has received less attention. We propose a methodology for evaluating an aggregated set of results. Our method elicits a relatively small number of human judgements for a given query and then uses these to facilitate a metric-based evaluation of any possible presentation for the query. An extensive user study with 13 verticals confirms that, when users prefer one presentation of results over another, our metric agrees with the stated preference. By using Amazon’s Mechanical Turk, we show that reliable assessments can be obtained quickly and inexpensively.

History

Publisher Statement

The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-642-20161-5_15

Date

01/04/2011

Exports

Exports