Gathering egocentric video and other sensor data with AAC users to inform narrative prediction

Personal narrative has long been accepted as a key part of our daily communication. Sharing our experiences with friends, family and others makes up a large proportion of our conversation [1]. Many of these narratives are told more than once: sharing the good news of a new job, the stressful story of the flat tyre in the highlands or all the things we experienced during our holidays. During a conversation, partners base their contributions amongst other factors on the context of the conversation, for example, who the conversation partner is or where and when a conversation takes place. Some AAC systems provide the user with phrases and whole narratives rather than simple word prediction. In a study by Todman et al. [2] handcrafted contextual conversational items were provided to AAC users on their device, communication rates of up to 64 wpm were demonstrated. However, using such a system requires hand scripted paragraphs and training users to remember the existence and location of these.
Automatic data-to-text sentence generators have been trialed in narrative based systems. In [3], a narrative ontol- ogy was populated with conversational topics linked to people and places. There are some attempts to use automatically gathered context information to improve prediction. Examples include providing vocabulary for ordering when located in a café or restaurant or presenting tagged pre-stored phrases depending on who the conversation partner is. Research by [4] has added topic identification to inform context aware SGDs but the implementation of a real-time system has yet to be achieved.

References: [1] Cheepen, C. (1988). The predictability of informal conversation. Oxford, Printer Publishers Ltd. [2] Todman J et al. 1995. Evaluation of the content of computer-aided conversations, AAC. [3] Dempster M et al. 2010. Automatic generation of conversational utterances and narrative for AAC: a prototype system. SLPAT. [4] Higginbotham DJ et al. 2012. The Application of Natural Language Processing to AAC, Assistive TECHNOLOGY. Evidence Area: AACcess emerging technologies, AACcess language and literacy, AACcess education, AACcess the community, AACcess employment, AACcess diversity, AACcess justice, AACcess culture, AACcess relationships, AACcess social media. Content Focus Area: Research Evidence.