Critical Questions Generation Leaderboard

Task

Critical Questions Generation is the task of automatically generating questions that can unmask the assumptions held by the premises of an argumentative text.

This leaderboard aims at benchmarking the capacity of language technology systems to create Critical Questions (CQs). That is, questions that should be asked in order to judge if an argument is acceptable or fallacious.

The task consists on generating 3 Useful Critical Questions per argumentative text.

All details on the task, the dataset, and the evaluation can be found in the paper Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models or in the Shared Task

Data

The CQs-Gen dataset gathers 220 interventions of real debates. Divided between:

  • validation: which contains 186 interventions and can be used for training or validation, as it has ~25 reference questions per intervention already evaluated accoding to their usefulness (either Useful, Unhelpful or Invalid).
  • test: which contains 34 interventions. The reference questions of this set (~70) are kept private to avoid data contamination. The questions generated using the test set are what should be submitted to this leaderboard.

Evaluation

The evaluation of each question is computer by comparing each of the 3 newly generated question to the reference questions of the test set using Semantic Text Similarity, and inheriting the label of the most similar reference given the threshold of 0.65. Questions where no reference is found are considered Invalid. See the evaluation function here, or find more details in the paper.

Leaderboard

Submissions

Results can be submitted for the test set only.

We expect submissions to be json files with the following format:

{
    "CLINTON_1_1": {
        "intervention_id": "CLINTON_1_1",
        "intervention": "CLINTON: "The central question in this election is really what kind of country we want to be and what kind of future we 'll build together
Today is my granddaughter 's second birthday
I think about this a lot
we have to build an economy that works for everyone , not just those at the top
we need new jobs , good jobs , with rising incomes
I want us to invest in you
I want us to invest in your future
jobs in infrastructure , in advanced manufacturing , innovation and technology , clean , renewable energy , and small business
most of the new jobs will come from small business
We also have to make the economy fairer
That starts with raising the national minimum wage and also guarantee , finally , equal pay for women 's work
I also want to see more companies do profit-sharing"",
        "dataset": "US2016",
        "cqs": [
            {
                "id": 0,
                "cq": "What does the author mean by "build an economy that works for everyone, not just those at the top"?"
            },
            {
                "id": 1,
                "cq": "What is the author's definition of "new jobs" and "good jobs"?"
            },
            {
                "id": 2,
                "cq": "How will the author's plan to "make the economy fairer" benefit the working class?"
            }
        ]
    },
...
}

After clicking 'Submit Eval' wait for a couple of minutes before trying to refresh.

If you find any issues, please email blanca.calvo@ehu.eus
Split