Detecting Argumentative Fallacies in the Wild: Problems and Limitations of Large Language Models

TY - GEN

T1 - Detecting Argumentative Fallacies in the Wild

T2 - Problems and Limitations of Large Language Models

AU - Ruiz-Dolz, Ramon

AU - Lawrence, John

N1 - Copyright: ©2023 Association for Computational Linguistics.

PY - 2023/12/7

Y1 - 2023/12/7

N2 - Previous work on the automatic identification of fallacies in natural language text has typically approached the problem in constrained experimental setups that make it difficult to understand the applicability and usefulness of the proposals in the real world. In this paper, we present the first analysis of the limitations that these data-driven approaches could show in real situations. For that purpose, we first create a validation corpus consisting of natural language argumentation schemes. Second, we provide new empirical results to the emerging task of identifying fallacies in natural language text. Third, we analyse the errors observed outside of the testing data domains considering the new validation corpus. Finally, we point out some important limitations observed in our analysis that should be taken into account in future research in this topic. Specifically, if we want to deploy these systems in the Wild.

AB - Previous work on the automatic identification of fallacies in natural language text has typically approached the problem in constrained experimental setups that make it difficult to understand the applicability and usefulness of the proposals in the real world. In this paper, we present the first analysis of the limitations that these data-driven approaches could show in real situations. For that purpose, we first create a validation corpus consisting of natural language argumentation schemes. Second, we provide new empirical results to the emerging task of identifying fallacies in natural language text. Third, we analyse the errors observed outside of the testing data domains considering the new validation corpus. Finally, we point out some important limitations observed in our analysis that should be taken into account in future research in this topic. Specifically, if we want to deploy these systems in the Wild.

UR - https://aclanthology.org/volumes/2023.argmining-1/

M3 - Conference contribution

SP - 1

EP - 10

BT - Proceedings of the 10th Workshop on Argument Mining (ArgMining 2023)

A2 - Alshomary, Milad

A2 - Chen, Chung-Chi

A2 - Muresan, Smaranda

A2 - Park, Joonsuk

A2 - Romberg, Julia

PB - Association for Computational Linguistics

CY - Stroudsburg, PA

ER -