About
ML Researcher with an academic background in Statistics and Mathematics. Currently researching evaluation-awareness and kinds of metagaming in language models, and how Inoculation prompting shapes learning.
Other things I’m thinking about:
Unverbalized evaluation awareness in language models: What techniques reliably catch unverbalized eval-awareness? IMO, it’s one of the biggest open challenges in relying on alignment evals.
- More broadly, my hypothesis is that CoT is going to be less and less helpful for any kind of monitoring as capabilities increase, eg. a spike in Astra’s no CoT capabilities, CoT obfuscation (situational awareness can make this worse).
Understanding generalization
- How language models generalize, specifically during RL-based posttraining and its implications for models’ safety.
Risks of Situational Awareness
- What are good solutions? Make models less situationally aware? Build better, more realistic evals to catch consequences before deployment? Develop training techniques that teach models to behave alike regardless of the current situation (eval vs deployment)? Train generally good models + design better specification strategies making sure models comply with developers’ intended training signal?
Other than my technical safety research interests, I like to learn about performance engineering and how computers work.
I have in mind a lot of open ideas and problems + concrete plans to tractably study many of them but I’m bottlenecked by compute funding. If you’re looking to collaborate, have funding for a technical safety problem you want to see progress on, I’m very interested in listening to ideas of all kinds. 👇