Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Downloads (Pure)

Abstract

In this paper, we perform a systematic analysis of how closely the intermediate layers from LSTM and trans former language models correspond to human semantic knowledge. Furthermore, in order to make more meaningful comparisons with theories of human language comprehension in psycholinguistics, we focus on two key stages where the meaning of a particular target word may arise: immediately before the word’s presentation to the model (comparable to forward inferencing), and immediately after the word token has been input into the network. Our results indicate that the transformer models are better at capturing semantic knowledge relating to lexical concepts, both during word prediction and when retention is required.
Original languageEnglish
Title of host publicationAssociation for Computational Linguistics
PublisherAssociation for Computational Linguistics (ACL)
Pages211
Number of pages221
Publication statusPublished - 01 Jun 2021
EventProceedings of the Workshop on Cognitive Modeling and Computational Linguistics - Mexico (Virtual), Mexico City, United States
Duration: 06 Jun 202111 Jun 2021
Conference number: 2021
https://2021.naacl.org/

Workshop

WorkshopProceedings of the Workshop on Cognitive Modeling and Computational Linguistics
Abbreviated titleCMCL/NNACL
Country/TerritoryUnited States
CityMexico City
Period06/06/202111/06/2021
Internet address

Keywords

  • Transformers
  • computational semantics
  • cognition
  • neural language models
  • LSTM

Fingerprint

Dive into the research topics of 'Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models'. Together they form a unique fingerprint.

Cite this