The CASS Technique for Evaluating the Performance of Argument Mining

Rory Duthie; John Lawrence; Katarzyna Budzynska; Chris Reed

doi:10.18653/v1/W16-2805

The CASS Technique for Evaluating the Performance of Argument Mining

Rory Duthie, John Lawrence, Katarzyna Budzynska, Chris Reed

Computing

Research output: Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

15 Citations (Scopus)

Abstract

Argument mining integrates many distinct computational linguistics tasks, and as a result, reporting agreement between annotators or between automated output and gold standard is particularly challenging. More worrying for the field, agreement and performance are also reported in a wide variety of different ways, making comparison between approaches difficult. To solve this problem, we propose the CASS technique for combining metrics covering different parts of the argument mining task. CASS delivers a justified
method of integrating results yielding confusion matrices from which CASS-κ and CASS-F1 scores can be calculated.

Original language	English
Title of host publication	Proceedings of the 3rd Workshop on Argument Mining (ArgMining2016)
Publisher	Association for Computational Linguistics
Pages	40-49
Number of pages	10
ISBN (Electronic)	9781945626173
DOIs	https://doi.org/10.18653/v1/W16-2805
Publication status	Published - Aug 2016
Event	3rd Workshop on Argument Mining - Berlin, Germany Duration: 12 Aug 2016 → 12 Aug 2016 http://argmining2016.arg.tech/

Conference

Conference	3rd Workshop on Argument Mining
Country/Territory	Germany
City	Berlin
Period	12/08/16 → 12/08/16
Internet address	http://argmining2016.arg.tech/

Access to Document

10.18653/v1/W16-2805

Argument Mining
Reed, C. (Investigator)
Engineering and Physical Sciences Research Council
1/01/16 → 31/12/19
Project: Research

Cite this

@inbook{c68f9d1d2e9d417da274f0a99189a012,

title = "The CASS Technique for Evaluating the Performance of Argument Mining",

abstract = "Argument mining integrates many distinct computational linguistics tasks, and as a result, reporting agreement between annotators or between automated output and gold standard is particularly challenging. More worrying for the field, agreement and performance are also reported in a wide variety of different ways, making comparison between approaches difficult. To solve this problem, we propose the CASS technique for combining metrics covering different parts of the argument mining task. CASS delivers a justifiedmethod of integrating results yielding confusion matrices from which CASS-κ and CASS-F1 scores can be calculated.",

author = "Rory Duthie and John Lawrence and Katarzyna Budzynska and Chris Reed",

year = "2016",

month = aug,

doi = "10.18653/v1/W16-2805",

language = "English",

pages = "40--49",

booktitle = "Proceedings of the 3rd Workshop on Argument Mining (ArgMining2016)",

publisher = "Association for Computational Linguistics",

url = "http://argmining2016.arg.tech/",

}

The CASS Technique for Evaluating the Performance of Argument Mining. / Duthie, Rory; Lawrence, John ; Budzynska, Katarzyna et al.
Proceedings of the 3rd Workshop on Argument Mining (ArgMining2016). Association for Computational Linguistics, 2016. p. 40-49 W16-2805.

Research output: Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

TY - CHAP

T1 - The CASS Technique for Evaluating the Performance of Argument Mining

AU - Duthie, Rory

AU - Lawrence, John

AU - Budzynska, Katarzyna

AU - Reed, Chris

PY - 2016/8

Y1 - 2016/8

N2 - Argument mining integrates many distinct computational linguistics tasks, and as a result, reporting agreement between annotators or between automated output and gold standard is particularly challenging. More worrying for the field, agreement and performance are also reported in a wide variety of different ways, making comparison between approaches difficult. To solve this problem, we propose the CASS technique for combining metrics covering different parts of the argument mining task. CASS delivers a justifiedmethod of integrating results yielding confusion matrices from which CASS-κ and CASS-F1 scores can be calculated.

AB - Argument mining integrates many distinct computational linguistics tasks, and as a result, reporting agreement between annotators or between automated output and gold standard is particularly challenging. More worrying for the field, agreement and performance are also reported in a wide variety of different ways, making comparison between approaches difficult. To solve this problem, we propose the CASS technique for combining metrics covering different parts of the argument mining task. CASS delivers a justifiedmethod of integrating results yielding confusion matrices from which CASS-κ and CASS-F1 scores can be calculated.

U2 - 10.18653/v1/W16-2805

DO - 10.18653/v1/W16-2805

M3 - Chapter (peer-reviewed)

SP - 40

EP - 49

BT - Proceedings of the 3rd Workshop on Argument Mining (ArgMining2016)

PB - Association for Computational Linguistics

T2 - 3rd Workshop on Argument Mining

Y2 - 12 August 2016 through 12 August 2016

ER -

The CASS Technique for Evaluating the Performance of Argument Mining

Abstract

Conference

Access to Document

Fingerprint

Projects

Argument Mining

Cite this