Loading Events

« All Events

  • This event has passed.

Ph.D.Thesis {Colloquium} ONLINE: CDS: 21st June 2021 : ” Relating Representations in Deep Neural Networks and the Brain”

21 Jun @ 2:00 PM -- 3:00 PM

Ph.D. Thesis Colloquium

Speaker                : Ms.  Sharmistha Jat

S.R. Number        : 06-18-02-10-12-15-1-13017

Title                      : “Relating Representations in Deep Neural Networks and the Brain“​
Date & Time        : June 21, 2021 (Monday), 02:00 PM
Venue                   : Online


Deep Neural Networks (DNN) inspired by the human brain have redefined the state-of-the-art performance in artificial intelligence during the past decade. Much of the research is still trying to understand and explain the function of these networks. In this thesis, we leverage knowledge from the neuroscience literature to evaluate the representations learned in state-of-the-art language models. We use sentences with simple syntax and semantics (e.g., “The bone was eaten by the dog.”), and train multiple neural networks to predict the part of speech, next word, as each word is read. We present other similar sentences word-by-word to humans in a magnetoencephalography (MEG) scanner. We subsequently train a linear regression model to predict observed brain recording from the hidden layers of the trained neural networks and some popular pre-trained networks like BERT and ELMo. Once trained, the linear model can detect stimulus (DNN feature representations) in test data (brain recordings) by calculating the accuracy of predicting brain activity from stimulus features; an above chance accuracy deems the stimulus decodable. Our results show that the middle layers of these networks are the most predictive of the recorded brain activity. But, a more fine-grained evaluation indicates that various types of stimuli (determiner, adjective, noun, verb) are represented more dominantly in different layers of the model.

Combining the meaning of small units to generate the meaning of the whole is known as semantic composition. We test the semantic composition capabilities of these DNN networks with respect to the human brain. Despite their impressive performance in benchmark tasks (such as SuperGLUE), it is often challenging to generalize DNNs to out-of-distribution inputs. This calls for a better evaluation of semantic composition in DNNs. Past research has used human-labeled judgments of meaning similarity to compare semantically composed representations at the phrase level. Instead of such proxy behavioral measures, in this work, we utilize brain Magnetoencephalography (MEG) data to evaluate semantic composition in DNNs against those in the brain at the sentence level. We perform this evaluation incrementally as each word in the sentence is processed in the brain and DNN. As a result, we can analyze the effect of the composition function in representing the same word as more of the sentence context becomes available. Our experiments show that DNN models perform above chance in predicting brain data, indicating good semantic composition. Furthermore, we find that the right frontal and right temporal brain regions have the best accuracy on our tests, rather than the language-dominant left brain regions. Previous research has suggested that these right-sided brain regions are responsible for executive and memory functions.
 As an additional contribution, we propose a new Dynamic Time Warping (DTW) based distance metric to evaluate alignment between the predicted brain activity versus the observed brain activity. This proposal allows us to account for neural data variability collected from a single subject.


21 Jun
2:00 PM -- 3:00 PM