A comprehensive multimodal dataset for contactless lip reading and acoustic analysis

TY - JOUR

T1 - A comprehensive multimodal dataset for contactless lip reading and acoustic analysis

AU - Ge, Yao

AU - Tang, Chong

AU - Li, Haobo

AU - Chen, Zikang

AU - Wang, Jingyan

AU - Li, Wenda

AU - Cooper, Jonathan

AU - Chetty, Kevin

AU - Faccio, Daniele

AU - Imran, Muhammad

AU - Abbasi, Qammer H.

N1 - © 2023. The Author(s).

PY - 2023/12/13

Y1 - 2023/12/13

N2 - Small-scale motion detection using non-invasive remote sensing techniques has recently garnered significant interest in the field of speech recognition. Our dataset paper aims to facilitate the enhancement and restoration of speech information from diverse data sources for speakers. In this paper, we introduce a novel multimodal dataset based on Radio Frequency, visual, text, audio, laser and lip landmark information, also called RVTALL. Specifically, the dataset consists of 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77 GHz frequency modulated continuous wave (FMCW) data from millimeter wave (mmWave) radar, visual and audio information, lip landmarks and laser data, offering a unique multimodal approach to speech recognition research. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words, and 16 sentences. The dataset has been validated and has potential for the investigation of lip reading and multimodal speech recognition.

AB - Small-scale motion detection using non-invasive remote sensing techniques has recently garnered significant interest in the field of speech recognition. Our dataset paper aims to facilitate the enhancement and restoration of speech information from diverse data sources for speakers. In this paper, we introduce a novel multimodal dataset based on Radio Frequency, visual, text, audio, laser and lip landmark information, also called RVTALL. Specifically, the dataset consists of 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77 GHz frequency modulated continuous wave (FMCW) data from millimeter wave (mmWave) radar, visual and audio information, lip landmarks and laser data, offering a unique multimodal approach to speech recognition research. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words, and 16 sentences. The dataset has been validated and has potential for the investigation of lip reading and multimodal speech recognition.

KW - Humans

KW - Lipreading

KW - Speech/physiology

KW - Speech Perception/physiology

KW - Radar

KW - Radio Waves

U2 - 10.1038/s41597-023-02793-w

DO - 10.1038/s41597-023-02793-w

M3 - Article

C2 - 38092796

SN - 2052-4436

VL - 10

JO - Scientific Data

JF - Scientific Data

IS - 1

M1 - 895

ER -