Significance testing for INEX 2008-09 Ad Hoc Track

Abstract

INEX (The INitiative for the Evaluation of XML retrieval) sponsors a competition that promotes the development/evaluation of XML-based retrieval systems. INEX provides a document collection, a query set (topics) and evaluation measures for use by the XML-based retrieval systems of the participants. We have developed a methodology for the retrieval of elements, at the appropriate level of granularity, within the XML document. This methodology is applied to the tasks of the INEX Ad Hoc Track. In this thesis, the focus lies on significance tests that are performed to see how our methods compare with those used by the top-ranked INEX participants. Our approach to significance testing is identical to that used by INEX for evaluating its participants. We use the results of each individual query; find the variance and standard deviation between the scores, and the value of t, which maps to probability. We use a confidence interval of 95% in one-tailed t-test (so the probability must be less than 0.05 to assure significance) to see if one method is significantly better than another. We evaluate results of the 2008 and 2009 Ad Hoc tasks using this approach. The results of these significance tests for the basic INEX Ad Hoc tasks are given in this paper along with observations on these results.