Abstract
In computational linguistics, the distributional hypothesis of word meaning has allowed us to construct distributional semantic models by mining large corpora of text to extract co-occurrence statistics of words. Neural approaches such as word2vec learn two sets of matrix representations from mined word-context counts, by using dot product between target and context vector with a sigmoid function to measure the probability of positive association. By using negative sampling techniques and gradient descent optimization, we can learn an approximation of word meaning. Our goal is to construct vector representations for a set of human-derived properties by using a neural topology, similar to the skip-gram word2vec, which uses the target word to predict the surrounding windowed context from mined statistics. Surveying a number of concepts for human interpretable features is costly and time-consuming, but unsupervised learning of vector space models from text data is cheap and accessible. We learn feature meaning by sampling a tiny subset of a pretrained set of word embeddings for which we know the properties. Negative sampling along with gradient descent applied only to the matrix of representations allows us to learn feature meaning in relation to the pretrained word vectors. In this case, a word and a feature have meaningful association if their vectors are close together, which we can measure using cosine similarity. Ranking these features for a given concept, we can extract salient features for the word. Furthermore, since these come from a wider vector space model, we can sample unseen words for features. The process allows us to extract possible feature of words, which could make further surveying the concepts for properties much faster. Active learning would then allow us to repeat this process with a larger lexicon which could be then surveyed again, this time with a higher probability of correctly sampling features.
Original language | English |
---|---|
Number of pages | 1 |
Publication status | Published - 02 Sep 2019 |
Event | International Conference of the Royal Statistical Society (RSS 2019) - Belfast, United Kingdom Duration: 02 Sep 2019 → 05 Sep 2019 https://events.rss.org.uk/rss/frontend/reg/thome.csp?pageID=83705&ef_sel_menu=1647&eventID=270 |
Conference
Conference | International Conference of the Royal Statistical Society (RSS 2019) |
---|---|
Abbreviated title | RSS 2019 |
Country/Territory | United Kingdom |
City | Belfast |
Period | 02/09/2019 → 05/09/2019 |
Internet address |
Fingerprint
Dive into the research topics of 'Feature2Vec: Distributional Semantic Modelling of Human Property Knowledge'. Together they form a unique fingerprint.Student Theses
-
Interpretable semantic representations from neural language models and computer vision
Author: Derby, S., Jul 2022Supervisor: Murphy, B. (Supervisor), Miller, P. (Supervisor) & Devereux, B. (Supervisor)
Student thesis: Doctoral Thesis › Doctor of Philosophy
File