Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

Steven Derby; Paul Miller; Barry Devereux

Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

Steven Derby, Paul Miller, Barry Devereux

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

69 Downloads (Pure)

Abstract

Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.

Original language	English
Title of host publication	Proceedings of the 28th International Conference on Computational Linguistics
Editors	Donia Scott, Nuria Bel, Chengqing Zong
Publisher	Association for Computational Linguistics
Pages	1906-1921
Number of pages	16
ISBN (Electronic)	9781952148279
Publication status	Published - 08 Dec 2020
Event	28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spain Duration: 08 Dec 2020 → 13 Dec 2020

Publication series

Name	COLING-International Conference on Computational Linguistics, Proceedings of the Conference

Conference

Conference	28th International Conference on Computational Linguistics, COLING 2020
Country/Territory	Spain
City	Virtual, Online
Period	08/12/2020 → 13/12/2020

Bibliographical note

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

ASJC Scopus subject areas

Computer Science Applications
Computational Theory and Mathematics
Theoretical Computer Science

Access to Document

Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks
Copyright 2020 the authors. This is an open access article published under a Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the author and source are cited.
Final published version, 11.7 MBLicence: CC BY

http://www.scopus.com/inward/record.url?scp=85149675586&partnerID=8YFLogxKLicence: Unspecified

Interpretable semantic representations from neural language models and computer vision
Derby, S. (Author), Murphy, B. (Supervisor), Miller, P. (Supervisor) & Devereux, B. (Supervisor), Jul 2022
Student thesis: Doctoral Thesis › Doctor of Philosophy

File

Barry Devereux
- B.Devereuxqub.acuk
- School of Electronics, Electrical Engineering and Computer Science - Senior Lecturer
- Institute of Electronics, Communications & Information Technology
Person: Academic

Cite this

Derby, S., Miller, P., & Devereux, B. (2020). Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. In D. Scott, N. Bel, & C. Zong (Eds.), Proceedings of the 28th International Conference on Computational Linguistics (pp. 1906-1921). (COLING-International Conference on Computational Linguistics, Proceedings of the Conference). Association for Computational Linguistics. http://www.scopus.com/inward/record.url?scp=85149675586&partnerID=8YFLogxK

Derby, Steven ; Miller, Paul ; Devereux, Barry. / Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. Proceedings of the 28th International Conference on Computational Linguistics. editor / Donia Scott ; Nuria Bel ; Chengqing Zong. Association for Computational Linguistics, 2020. pp. 1906-1921 (COLING-International Conference on Computational Linguistics, Proceedings of the Conference).

@inproceedings{026fbe37744843f09fceb3059532abf7,

title = "Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks",

abstract = "Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.",

author = "Steven Derby and Paul Miller and Barry Devereux",

note = "Publisher Copyright: {\textcopyright} 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.; 28th International Conference on Computational Linguistics, COLING 2020 ; Conference date: 08-12-2020 Through 13-12-2020",

year = "2020",

month = dec,

day = "8",

language = "English",

series = "COLING-International Conference on Computational Linguistics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics",

pages = "1906--1921",

editor = "Donia Scott and Nuria Bel and Chengqing Zong",

booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",

}

Derby, S, Miller, P & Devereux, B 2020, Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. in D Scott, N Bel & C Zong (eds), Proceedings of the 28th International Conference on Computational Linguistics. COLING-International Conference on Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics, pp. 1906-1921, 28th International Conference on Computational Linguistics, COLING 2020, Virtual, Online, Spain, 08/12/2020. <http://www.scopus.com/inward/record.url?scp=85149675586&partnerID=8YFLogxK>

Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. / Derby, Steven; Miller, Paul; Devereux, Barry.
Proceedings of the 28th International Conference on Computational Linguistics. ed. / Donia Scott; Nuria Bel; Chengqing Zong. Association for Computational Linguistics, 2020. p. 1906-1921 (COLING-International Conference on Computational Linguistics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

AU - Derby, Steven

AU - Miller, Paul

AU - Devereux, Barry

PY - 2020/12/8

Y1 - 2020/12/8

N2 - Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.

AB - Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.

M3 - Conference contribution

AN - SCOPUS:85149675586

T3 - COLING-International Conference on Computational Linguistics, Proceedings of the Conference

SP - 1906

EP - 1921

BT - Proceedings of the 28th International Conference on Computational Linguistics

A2 - Scott, Donia

A2 - Bel, Nuria

A2 - Zong, Chengqing

PB - Association for Computational Linguistics

T2 - 28th International Conference on Computational Linguistics, COLING 2020

Y2 - 8 December 2020 through 13 December 2020

ER -

Derby S, Miller P, Devereux B. Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. In Scott D, Bel N, Zong C, editors, Proceedings of the 28th International Conference on Computational Linguistics. Association for Computational Linguistics. 2020. p. 1906-1921. (COLING-International Conference on Computational Linguistics, Proceedings of the Conference).

Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Fingerprint

Student theses

Interpretable semantic representations from neural language models and computer vision

Profiles

Barry Devereux

Cite this