I focus on computational models of prediction in sentence processing. I'm interested in the interaction between language-specific information and domain-general cognitive processes like memory.
Recently, I've been looking at ways in which experimental data from humans compares with predictions made by large language models, in the service of developing more cognitively plausible models of language processing.
I'm also interested how cognitive modeling can interact with other NLP applications, such as interpetability and computational social sciences.
In general, I'm excited about various topics at the intersection of language, computation, and cognition, and am always happy to connect regarding either of these interests.
Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?
Findings of EMNLP 2023
An important assumption that comes with using LLMs on psycholinguistic data has gone unverified. LLM-based predictions are based on subword tokenization, not decomposition of words into morphemes.
Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data.
Our results replicate previous findings. In the aggregate, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation.
Finer-grained analyses show differences between BPE and morphological surprisal, suggesting that the effects of subword tokenization should be taken seriously in cognitive research, and provide promising evidence for using morphological surprisal in further research.
How far does probability take us when measuring psycholinguistic fit? Evidence from substitution illusions and speeded cloze data
Poster Presentation at the 36th Annual Conference on Human Sentence Processing, 2023
When asked How many animals of each kind did Moses take on the ark?, humans often respond with two instead of detecting issues with the question.
This effect is known as a substitution illusion (canonically a Moses illusion, but we refer to them this way to emphasize how they can apply in various contexts).
We used measures from NLP techniques on experimental data collected by Muller (2022) to address whether people's selectivity to the illusion is better explained by semantic similarity of the correct word and substituted word (distributional vector-space representations), or the substituted word's relationship to the context(masked language modeling).
On a separate speeded cloze task (Staub et al, 2015), we found that language model probability corresponded strongly with cloze probability, but not with reaction time, which is seen as a stronger measure of lexical predictability in context.
This work reveals some limitations of applying language models to psycholinguistic data.
Contextualized Word Embeddings Capture Human-Like Relations Between English Word Senses
Oral Presentation for Cognitive Aspects of the Lexicon workshop(CogALex VI) at International Conference on Computational Linguistics (COLING), 2020
Undergraduate Honors Thesis, advised by Dr. Meylan, Prof. Srinivasan, and Prof. Steven Piantadosi, received the Glushko Prize for Outstanding Undergraduate Research in Cognitive Sciences.
Code and Data
We investigated whether recent advances in NLP (specifically the Transformer-based neural network model BERT), are able to capture human-like distinctions between
meanings of the same word, such as polysemy and homonymy. We collected human judgements of the relatedness of selected WordNet senses for 32 English words
from a two-dimensional spatial arrangement task, and compare them with relatedness according to BERT vectors for these corresponding senses in the SemCor corpus.
We demonstrated participants’ judgments of the relatedness between senses are correlated with distances between senses in the BERT embedding space, and that BERT encodes
homonymous sense relations closer to human judgements than polysemous ones.
Evaluating Models of Robust Word Recognition with Serial Reproduction.
Published in May 2021 issue of Cognition journal
Journal Article |
Preprint (full text)
We compared how several probabilistic generative language models, such as n-grams, probabilistic context free grammars (PCFGs), and neural networks,
capture human linguistic expectations in a web-based serial reproduction task, in which in which participants try to repeat sentences said by other participants,
similar to a game of "Telephone." We found that models that make use of preceding context, especially those with abstract representations of linguistic structure, best predict changes participants
made when trying to reproduce utterances in the experiment. I contributed to designing and implementing parts of the experimental interface,
extracting probabilities under PCFGs, modeling which words in utterances were most likely to change under the models, and revising the final paper.
Collaborators, Mentors, Friends, and other Co-Conspirators
Research is never done in a vacuum, and publications don't reflect everyone who's intellectually influenced me. Here are some of those people.
- UMD Cohort: Sebastián Mancha, Utku Türk, Sarah Boukendour,
Cassandra Caragine, Lydia Quevedo, Allison Dods,
Malhaar Shah, Sathvik Nair (SUS CLAMS for short)
- Labmates: Rupak Sarkar, Alexander Hoyle, Dr. Pranav Goel, London Dixon,
Katherine Howitt, Rosa Lee, Dr. Masato Nakamura, Allison MacDonald,
Joselyn Rodriguez, Nika Jurov, Dr. Craig Thorburn, Neha Srikanth, Maharshi Gor, Yu (Hope) Hou
- Collaborators and Mentors: Jess Mankewitz, Dr. Sammy Floyd, Dr. Mika Braginsky, Prof. Naomi Feldman, Prof. Ellen Lau,
Prof. Shohini Bhattasali, Dr. Stephan Meylan, Dr. Ruthe Foushee, Prof. Cassandra Jacobs
Other projects (not just academic) and information.
- Large Language Models & Levels of Analysis: What 40-year-old Neuroscience Research can Tell Us About Modern AI
– Applying fundamental ideas from cognitive science (Marr's levels of analysis) to explain various kinds of language models and argue how we can't say LLMs "understand" language like humans do.
- childes-db - I helped update & refactor the implementation of a relational database interface to CHILDES, a collection of multilingual child language corpora, so it can be easily accessed in Python and R.
- GPT-3: An AI Breakthrough, but not Coming for Your Job
– Article describing GPT-3, reactions from the press and experts, and research-backed opinions on the technology's limitations for Skynet Today (AI news publication). Coauthored with Daniel Bashir
- How Biases in Language get Perpetuated by Technology
– Towards Data Science article on personal project investigating gender, racial, and religious bias through analogy evaluation with static word embeddings (GloVe)
- Workshop on NLP/ML– given at the Spectra 3.0 hackathon.
Presented overview of the field and sentiment classification demo on tweets related to mental health.
- Letters for Black Lives- I was involved with writing & curating resources for the South Asian community on anti-Blackness, including Hindi translation.
- Here are some organizations whose work I care about: DMV Mutual Aid, Kalama Mutual Aid, Queer in AI, Bay Area Solidarity Summer, Sogorea Te' Land Trust
- In my spare time, I enjoy playing violin & South Asian percussion, cooking, going on runs and hikes, and most recently, exploring the East Coast.