Robust Audio-Visual Speech Recognition under Noisy Audio-Video Conditions

D. Stewart, R. Seymour, A. Pass, Ming Ji

Research output: Contribution to journalArticlepeer-review

60 Citations (Scopus)
703 Downloads (Pure)


This paper presents the maximum weighted stream posterior (MWSP) model as a robust and efficient stream integration method for audio-visual speech recognition in environments, where the audio or video streams may be subjected to unknown and time-varying corruption. A significant advantage of MWSP is that it does not require any specific measurements of the signal in either stream to calculate appropriate stream weights during recognition, and as such it is modality-independent. This also means that MWSP complements and can be used alongside many of the other approaches that have been proposed in the literature for this problem. For evaluation we used the large XM2VTS database for speaker-independent audio-visual speech recognition. The extensive tests include both clean and corrupted utterances with corruption added in either/both the video and audio streams using a variety of types (e.g., MPEG-4 video compression) and levels of noise. The experiments show that this approach gives excellent performance in comparison to another well-known dynamic stream weighting approach and also compared to any fixed-weighted integration approach in both clean conditions or when noise is added to either stream. Furthermore, our experiments show that the MWSP approach dynamically selects suitable integration weights on a frame-by-frame basis according to the level of noise in the streams and also according to the naturally fluctuating relative reliability of the modalities even in clean conditions. The MWSP approach is shown to maintain robust recognition performance in all tested conditions, while requiring no prior knowledge about the type or level of noise.

Original languageEnglish
Article number6495474
Pages (from-to)175-184
Number of pages10
JournalIEEE Transactions on Cybernetics
Issue number2
Early online date08 Apr 2013
Publication statusPublished - 13 Jan 2014

Bibliographical note

INSPEC Accession Number:
Date of publication April 8, 2013; date of current version
January 13, 2014


Dive into the research topics of 'Robust Audio-Visual Speech Recognition under Noisy Audio-Video Conditions'. Together they form a unique fingerprint.

Cite this