Disagreement space in argument analysis

Annette Hautli-Janisz, Ella Schad, Chris Reed

Research output: Chapter in Book/Report/Conference proceedingConference contribution

60 Downloads (Pure)


For a highly subjective task such as recognising speaker intention and argumentation, the traditional way of generating gold standards is to aggregate a number of labels into a single one. However, this seriously neglects the underlying richness that characterises discourse and argumentation and is also, in some cases, straightforwardly impossible. In this paper, we present QT30nonaggr, the first corpus of non-aggregated argument annotation. QT30nonaggr encompasses 10% of QT30, the largest corpus of dialogical argumentation and analysed broadcast political debate currently available with 30 episodes of BBC’s ‘Question Time’ from 2020 and 2021. Based on a systematic and detailed investigation of annotation judgements across all steps of the annotation process, we structure the disagreement space with a taxonomy of the types of label disagreements in argument annotation, identifying the categories of annotation errors, fuzziness and ambiguity.

Original languageEnglish
Title of host publicationProceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022
EditorsGavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, Alexandra Uma
Place of PublicationParis
PublisherEuropean Language Resources Association (ELRA)
Number of pages9
ISBN (Electronic)9791095546986
Publication statusPublished - 2022
Event1st Workshop on Perspectivist Approaches to NLP - Marseille, France
Duration: 20 Jun 202220 Jun 2022


Conference1st Workshop on Perspectivist Approaches to NLP
Abbreviated titleNLPerspectives 2022


  • argumentation and conflict
  • broadcast political debate
  • Inference Anchoring Theory
  • Question Time

ASJC Scopus subject areas

  • Information Systems and Management
  • Information Systems
  • Computer Science Applications


Dive into the research topics of 'Disagreement space in argument analysis'. Together they form a unique fingerprint.

Cite this