Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Distributional models provide a convenient way to model semantics using dense embedding spaces derived from unsupervised learning algorithms. However, the dimensions of dense embedding spaces are not designed to resemble human semantic knowledge. Moreover, embeddings are often built from a single source of information (typically text data), even though neurocognitive research suggests that semantics is deeply linked to both language and perception. In this paper, we combine multimodal information from both text and image-based representations derived from state-of-the-art distributional models to produce sparse, interpretable vectors using Joint Non-Negative Sparse Embedding. Through in-depth analyses comparing these sparse models to human-derived behavioural and neuroimaging data, we demonstrate their ability to predict interpretable linguistic descriptions of human ground-truth semantic knowledge.

Original languageEnglish
Title of host publicationCoNLL 2018 - 22nd Conference on Computational Natural Language Learning, Proceedings
PublisherAssociation for Computational Linguistics (ACL)
Pages260-270
Number of pages11
ISBN (Electronic)9781948087728
Publication statusPublished - 01 Jan 2018
Event22nd Conference on Computational Natural Language Learning, CoNLL 2018 - Brussels, Belgium
Duration: 31 Oct 201801 Nov 2018

Conference

Conference22nd Conference on Computational Natural Language Learning, CoNLL 2018
CountryBelgium
CityBrussels
Period31/10/201801/11/2018

ASJC Scopus subject areas

  • Linguistics and Language
  • Artificial Intelligence
  • Human-Computer Interaction

Fingerprint Dive into the research topics of 'Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge'. Together they form a unique fingerprint.

  • Cite this

    Derby, S., Miller, P., Murphy, B., & Devereux, B. (2018). Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge. In CoNLL 2018 - 22nd Conference on Computational Natural Language Learning, Proceedings (pp. 260-270). Association for Computational Linguistics (ACL).