Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model

Steven Derby, Paul Miller, Brian Murphy, Barry Devereux

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)
30 Downloads (Pure)

Abstract

Performance in language modelling has been significantly improved by training recurrent neural networks on large corpora. This progress has come at the cost of interpretability and an understanding of how these architectures function, making principled development of better language models more difficult. We look inside a state-of-the-art neural language model to analyse how this model represents high-level lexico-semantic information. In particular, we investigate how the model represents words by extracting activation patterns where they occur in the text, and compare these representations directly to human semantic knowledge.

Original languageEnglish
Title of host publicationProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
PublisherAssociation for Computational Linguistics
Pages362-364
Number of pages3
ISBN (Electronic)9781948087711
DOIs
Publication statusPublished - Nov 2018
Event1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 - Brussels, Belgium
Duration: 01 Nov 2018 → …

Publication series

NameEMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,
PublisherAssociation for Computational Linguistics

Conference

Conference1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Country/TerritoryBelgium
CityBrussels
Period01/11/2018 → …

Bibliographical note

Funding Information:
This work was partly funded by a Microsoft Azure for Research Award.

Publisher Copyright:
© 2018 Association for Computational Linguistics

ASJC Scopus subject areas

  • Computer Science Applications
  • Computational Theory and Mathematics
  • Information Systems

Fingerprint

Dive into the research topics of 'Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model'. Together they form a unique fingerprint.

Cite this