Average Position of User Clicks as an Automated and Non-Intrusive Way of Evaluating Ranking Methods
2003-02-05
Loading...
View/Download File
Persistent link to this item
Statistics
View StatisticsJournal Title
Journal ISSN
Volume Title
Title
Average Position of User Clicks as an Automated and Non-Intrusive Way of Evaluating Ranking Methods
Published Date
2003-02-05
Publisher
Type
Report
Abstract
The need for an objective and automated way of evaluating the performance of different ranking/reranking methods is becoming increasingly important in the web search/meta search domain. There are various methods for ranking search results ranging from traditional information retrieval approaches to more recent methods based on link analysis and other quality measures that can be derived from the documents. There are also a number of strategies for combining different heuristics and answers from multiple experts. With all of these possibilities it is becoming increasingly difficult to find the best parameters, the best method, or the best mixture of methods that will maximize the quality for a particular query type or domain. This paper addresses the problem of automatically comparing the quality of the ordering of documents that are presented to the user as a sorted list according to believed relevance for a given topic or query. We introduce the average position of user clicks metric as an implicit, automated, and non-intrusive way of evaluating ranking methods. We also discuss under which situations and assumptions this metric can be used objectively by addressing various bias sources. Experiments performed in our meta search engine suggests that, this approach has the potential to sample a wide range of query types and users with greater statistical significance compared to methods that rely on explicit user judgements.
Keywords
Description
Related to
Replaces
License
Series/Report Number
Funding information
Isbn identifier
Doi identifier
Previously Published Citation
Suggested citation
Uygar Oztekin, B.; Karypis, George; Kumar, Vipin. (2003). Average Position of User Clicks as an Automated and Non-Intrusive Way of Evaluating Ranking Methods. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/215552.
Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.