Following the success of ResPubliQA 2009, we are pleased to announce ResPubliQA 2010, the second evaluation campaign of Question Answering system over European Legislation to be held within the framework CLEF 2010 conference.
ResPubliQA 2010: TASK OVERVIEW
The aim of ResPubliQA 2010 is to capitalize on what has been achieved in the previous evaluation campaign while at the same time adding a number of refinements:
- The addition of new question types and the refinement of old ones;
- The opportunity to return both paragraph and exact answer;
- The addition of a new collection: EUROPARL
Two separate tasks are proposed for the ResPubliQA 2010 evaluation campaign:
1. PARAGRAPH SELECTION (PS) TASK: to retrieve one paragraph containing the answer to a question in natural language. One of the following responses must be returned:
a) ONE single paragraph containing the candidate answer
b) the string NOA to indicate that the system prefers not to answer the question.
2. ANSWER SELECTION (AS) TASK: beyond retrieving a paragraph containing the answer to a question in natural language, systems are required to demarcate also the exact answer. One of the following responses must be returned:
a) the exact answer highlighted inside one paragraph
b) the string NOA to indicate that the system prefers not to answer the question.
N.B. Systems that prefer to leave some questions unanswered, can OPTIONALLY decide to submit also a candidate paragraph/answer with the aim of evaluating the validation performance.
The two tasks are only different in the output required. Document collection and test data for both tasks are the same.
The results of the evaluation campaign will be discussed in a 1/2 day workshop as a CLEF 2010 Lab in Padova (Italy), 22 or 23 September 2010.
