Attribute-Enhanced Face Recognition with Neural Tensor Fusion Networks

Guosheng Hu, Yang Hua, Yangyuan Yuan, Zhihong Zhang, Sankha Subhra Mukherjee, Timothy Hospedales, Neil Robertson, Yongxin Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

60 Citations (Scopus)
560 Downloads (Pure)


Deep learning has achieved great success in face recognition, however deep-learned features still have limited invariance to strong intra-personal variations such as large
pose changes. It is observed that some facial attributes (e.g. eyebrow thickness, gender) are robust to such variations. We present the first work to systematically explore how the fusion of face recognition features (FRF) and facial attribute features (FAF) can enhance face recognition performance in various challenging scenarios. Despite the promise of FAF, we find that in practice existing fusion methods fail to leverage FAF to boost face recognition performance in some challenging scenarios. Thus, we develop a powerful tensor-based framework which formulates feature fusion as a tensor optimisation problem. It is nontrivial to directly optimise this tensor due to the large number of parameters to optimise. To solve this problem, we establish a theoretical equivalence between low-rank tensor optimisation and a two-stream gated neural network. This equivalence allows tractable learning using standard neural network optimisation tools, leading to accurate and stable optimisation. Experimental results show the fused feature works better than individual features, thus proving for the first time that facial attributes aid face recognition. We achieve state-of-the-art performance on three popular databases: MultiPIE (cross pose, lighting and expression), CASIA NIR-VIS2.0 (cross-modality environment) and LFW (uncontrolled environment).
Original languageEnglish
Title of host publicationInternational Conference on Computer Vision (ICCV) 2017: Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages10
ISBN (Electronic)978-1-5386-1032-9
ISBN (Print)978-1-5386-1033-6
Publication statusPublished - 25 Dec 2017
EventInternational Conference on Computer Vision 2017 - Venice, Italy
Duration: 22 Oct 201729 Oct 2017

Publication series

NameIEEE International Conference on Computer Vision (ICCV)
ISSN (Print)2380-7504


ConferenceInternational Conference on Computer Vision 2017


Dive into the research topics of 'Attribute-Enhanced Face Recognition with Neural Tensor Fusion Networks'. Together they form a unique fingerprint.

Cite this