Abstract
Argumentation has played a fundamental role in society for centuries from debate in the public sphere to everyday conversation. Most recently computational research into argumentation has focussed upon argument mining, the automatic extraction of reasoning structures from natural language. Although the content of argument (logos) is a funda- mental part of persuasion, ethos, the character of the speaker, also plays a significant role in communication as one of Aristotles modes of persuasion. This is identified within both argumentation (e.g. argument from expert opinion) and within conversation and debate, as sometimes a stronger character outweighs logical reasoning. Despite the importance of ethos, it has not been considered computationally in a solitary sense rather than as a bi-product of argumentation schemes or when performing argument mining on debate.This body of research aims to perform the novel annotation of ethos, as the sole phenomenon of focus, and develop a set of ethos technologies showcasing the importance of ethos in relation to political events through ethos mining. Due to its consistent debate structure and the large amount of available historical data, Hansard (the UK parliamentary debate record) is used as an exploratory domain. Four research questions were then formulated as the objectives of this PhD project: (RQ1) Can ethos be reliably annotated in natural language? (RQ2) Can ethos be reliably extracted automatically from natural language? (RQ3) Can fine-grained ethos types used by speakers be reliably annotated and automatically extracted? (RQ4) Can the analysis of appeals to ethos give an insight into the dynamics of the political landscape through the interactions between politicians?
Following Aristotle, ethos is specified as a property of an identifiable individual or identifiable group of individuals. Two types of ethotic linguistic activities are considered. The individual or individuals can be supported by others, e.g. “Mr. John Moore said, My hon. Friend is assiduously pursuing his constituents’ interests” (positive ethotic statement) or they can be attacked by others, e.g. “Mr. Bruce Grocott said, Is it not the simple truth that the Government are making the country sick?”, (negative ethotic statement).
In order to answer research questions RQ1 and RQ2, two natural language processing (NLP) pipelines, a rule-based and a deep learning approach (RQ2), were designed to automatically mine ethos in a manually annotated corpus (RQ1). To answer RQ3 a further NLP pipeline, building upon the supports and attacks of ethos extracted for RQ1 and RQ2, was created with the goal of differentiating ethos types, the grounds for which a speaker attacks or supports ethos. Finally, to answer RQ4 the output from RQ1 and RQ2 of positive and negative ethotic statements were used to produce ethos analytics based upon counts of supports and attacks for individual politicians, comparisons with external publications and relationships between politicians.
In the first approach to answer RQ1 and RQ2, this body of research produced the first corpus of ethos supports and attacks grounded in the rhetoric theory and focussing solely on ethos using transcripts from Hansard for manual annotation. The corpus incorporated 60 transcripts overall (70,117 words in total) annotated at a sentence level to allow for the full context of an ethotic support or attack to be captured. An inter annotator agreement study gave a Cohen’s kappa of κ =0.67 on the identification of ethos, when determining support or attack κ =0.95, κ =1 for determining the source speaker and κ =0.84 for determining the target speaker. This result shows the reliability of the annotation guidelines and thus provides a positive outcome in regard to RQ1. Following this was the first rule-based automatic extraction of ethos, trained and tested on data from the manually annotated corpus addressing RQ2. The pipeline of standard NLP techniques and domain specific rules developed specifically for ethos mining gave an F1-score of 0.70 (53% above baseline) when determining if a sentence contains ethos and a macro-averaged F1-score of 0.78 (16% above baseline) when determining the polarity (positive or negative). The evaluation of the rule-based approach indicates reliable extraction results delivering RQ2.
To aid in generalisation this process was repeated. For RQ1, a further thirty transcripts from Hansard were manually annotated for training data and the 60 transcripts from the first iteration were re-annotated to a new set of annotation guidelines, ensuring ethos is present on the surface of a sentence, resulting in a total of 90 transcripts. The improved guidelines gave a Cohen’s kappa of κ =0.67 on the identification of ethos, when determ- ining support or attack κ =1, κ =1 for determining the source speaker and κ =0.93 for determining the target speaker. This evaluation shows a reliable annotation in line with RQ1, in particular consistency when determining ethotic statements and improvements on identifying the polarity and target of each sentence. For RQ2, deep learning methods from image classification were developed for the novel application in text classification. A Deep Modular Recurrent Neural Network (DMRNN) was created to automatically identify ethotic statements and then determine their polarity making use of a NLP pipeline. The DMRNN and full pipeline gave an F1-score of 0.74 (21% above baseline and 6% when compared with the rule-based approach) for identifying ethotic statements and a macro-averaged F1-score of 0.84 (31% above baseline and 8% when compared with the first approach) for determining if an ethotic statement is positive or negative indicating that deep learning is reliable for the automatic extraction of ethos in the case of RQ2.
Addressing RQ3, all 90 transcripts were further annotated using the Aristotelian distinction of elements of ethos, Wisdom, Virtue and Goodwill (wisdom referring to practical experience, virtue to character traits and goodwill to aligning with the audience). An annotation evaluation on a 10% subset of the data gave an average Cohen’s κ of 0.52. The results, defined as moderate agreement (Landis and Koch, 1977) show the reliability of annotation and yet the difficulty of the task at hand which can be improved through further annotation iterations. An automatic classification using pairwise classifiers and one versus all classification gave F1-scores averaging 0.62 showing room for improvement.
Three applications of ethos were identified in relation to RQ4: graph analytics, qualitative analytics and quantitative analytics. The output from RQ2, both for the rule-based and deep learning approaches, allowed for the analysis of ethotic relations between politicians and political groupings to give an insight into the political landscape. Firstly, this shows which politicians attacked or supported one another as a directed graph. Secondly, qualitative analytics were developed exploring time series data for supports and attacks on individual politicians enabling the investigation of correlations with political events such as individual party position appointments. Finally, quantitative analytics were developed exploring time series data for supports and attacks on political parties enabling the investigation of correlations to general election results. Each of these steps give a visual, qualitative and quantitative insight into the political landscape of the time addressing RQ4. In summary, this research has described novel advances in the new sub-field of argument mining, ethos mining, contributing: (i) manual corpora of ethos supports and attacks; (ii) the automatic classification of ethos supports and attacks; (iii) the creation of deep learning methods (DMRNN) in text classification for extracting ethos; (iv) manually annotated corpora containing ethos types; (v) the development of an NLP pipeline for classifying ethos types; and (vi) a set of ethos analytics. These advances are widely applicable to various domains not only as a tool to gauge political opinion, but to extract public opinion from various sources of natural language. Social media discussions or public deliberation can be used to build profiles of individuals over large time periods from the natural language and identify individuals or groups at the centre of popular (or unpopular) opinion.
| Date of Award | 2020 |
|---|---|
| Original language | English |
| Sponsors | Engineering and Physical Sciences Research Council |
| Supervisor | Katarzyna Budzynska (Supervisor) & Chris Reed (Supervisor) |