Visual and Semantic Feature Spaces for Zero-shot Image Decoding

Large image galleries are notoriously difficult to categorise and navigate without a great deal of manual organisation. We present a new methodology that could enable a Brain-Computer Interface (BCI) system for image retrieval, as an alternative to the automated tagging approaches recently emerging. We propose a regression framework that allows us to effectively map EEG signals into a visuo-semantic feature space, where the stimulus image can be matched. We use a set of intermediate features ranging from low-level edge detection, to the semantic properties of an image’s depicted content, reflecting theoretical and empirical knowledge of how the brain processes visual input and extracts meaning from it. A real-world system would need to generalise to images not seen during training, so we use zero-shot learning to decode image features from brain activity. Using our approach under this challenging setting, we were able to decode left-out individual stimuli at a rate significantly above chance.