Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models

Research output: Chapter in Book/Report/Conference proceedingConference contribution

59 Downloads (Pure)

Abstract

Researchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplexity on sequence learning tasks such as language modelling. Such a procedure is possible due to the design of the softmax classification layer, which previous work has shown to comprise a viable set of semantic representations for the model vocabulary, and these these output embeddings are known to perform well on word similarity benchmarks. In this paper, we make meaningful comparisons between the input and output embeddings and other SOTA distributional models to gain a better understanding of the types of information they represent. We also construct a new set of word embeddings using the output embeddings to create locally-optimal approximations for the intermediate representations from the language model. These locally-optimal embeddings demonstrate excellent performance across all our evaluations.
Original languageEnglish
Title of host publicationProceedings of the 24th Conference on Computational Natural Language Learning
Pages442-454
Number of pages13
Publication statusPublished - 16 Nov 2020

Fingerprint Dive into the research topics of 'Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models'. Together they form a unique fingerprint.

Cite this