Publications

After testing 23 LLMs from 8 model series, we find that larger LLMs are more vulnerable to poisoning. Our jailbreak-tuning method causes GPT-4o to capably answer virtually any harmful question. We investigated the vulnerability to three forms of data poisoning: malicious fine-tuning, imperfect data curation, and intentional data contamination.

While larger language models exhibit impressive capabilities, they remain vulnerable to adversarial prompts. Empirical findings show that robustness against such attacks significantly improves with adversarial training, but not with model scaling alone.

To understand how neural networks generalize, we studied an RNN trained to play Sokoban. The RNN learned to plan ahead by ‘pacing’ despite penalties, demonstrating that reinforcement learning can encourage strategic planning in neural networks.

Evaluating three neural network circuits (IOI, greater-than, and docstring) under adversarial conditions reveals that the IOI and docstring circuits fail to match the full model’s behavior even on benign inputs, underscoring the need for more robust circuits in safety-critical applications.

Reinforcement learning from human feedback (RLHF) uses KL divergence regularization to mitigate reward errors, allowing high utility with light-tailed errors but suffering from reward hacking with heavy-tailed errors. While current models have light-tailed errors, real-world applications may still face significant risks from heavy-tailed errors, leading to catastrophic Goodhart.

InterpBench is a collection of 17 semi-synthetic transformers with known circuits, trained using Strict Interchange Intervention Training (SIIT). These models exhibit realistic weights and activations that reflect ground truth circuits, providing a valuable benchmark for evaluating mechanistic interpretability techniques.

By adapting existing interpretability techniques to the Mamba architecture, we partially reverse-engineered the circuit responsible for the Indirect Object Identification task, identifying layer 39 as a key component, and demonstrating the potential of these techniques to generalize to new architectures.

Existing circuits in the mechanistic interpretability literature may not be as faithful as reported. Current circuit faithfulness scores reflect both the methodological choices of researchers and the actual components of the circuit.

Guaranteed safe (GS) AI ensures AI systems avoid harmful behaviors through a world model, safety specification, and verifier to provide high-assurance safety guarantees. The paper discusses creating these components, technical challenges, and argues for the necessity and superiority of this approach to AI safety.

STARC (STAndardised Reward Comparison) metrics, a class of pseudometrics, quantify differences between reward functions, providing theoretical and empirical tools to improve the analysis and safety of reward learning algorithms in reinforcement learning.

Do language models implicitly learn a concept of human wellbeing? We explore this through the ETHICS Utilitarianism task, assessing if scaling enhances pretrained models’ representations.

Language model attacks typically assume one of two extreme threat models: full white-box access to model weights, or black-box access …

We found a way to modify neural networks to make their internals more interpretable and steerable while causing only a small degradation of performance. At each layer, we apply a quantization bottleneck that forces the activation vector into a sum of a few discrete codes; converting an inscrutable, dense, and continuous vector into a discrete list of codes from a learned codebook that are either on or off.

We show how to use Vision-Language Models (VLM), and specifically CLIP models, as reward models (RM) for RL agents. Instead of manually specifying a reward function, we only need to provide text prompts like ‘a humanoid robot kneeling’ to instruct and provide feedback to the agent. Importantly, we find that larger VLMs provide more accurate reward signals, so we expect this method to work even better with future models.

This paper introduces a statistical method for eliciting beliefs encoded in LLMs using surveys. We apply this method to study the encoded moral beliefs in 28 open- and closed-source LLMs. Our results demonstrate that most LLMs exhibit low uncertainty in unambiguous moral scenarios and that their preferences align with common sense judgements. In ambiguous moral scenarios, we find that only a few LLMs exhibit clear preferences and that closed-source models tend to agree with each other.

This paper systematizes the mechanistic interpretability process into 3 iterative steps, then proceeds to automate one of them: circuit discovery. Two of the algorithms presented automatically discover interpretability results previously established by human inspection.

This paper presents instances of inverse scaling: tasks where language models get worse with scale rather than better. These 11 examples were selected from 99 submissions in an open competition, the Inverse Scaling Prize. The paper also discusses inverse scaling in the literature and identifies four potential causes of inverse scaling. The prize-winning tasks have helped drive the discovery of U-shaped and inverted-U scaling trends, where an initial trend reverses, suggesting that scaling trends are less reliable at predicting the behavior of larger-scale models than previously understood.

The paper presents a new algorithm called Imitation learning from Language Feedback (ILF) that enables pre-trained large language models to learn from natural language feedback at training time, which is both user-friendly and sample-efficient. The algorithm can be seen as minimizing the KL divergence to the ground truth distribution. The paper shows that ILF outperforms both fine-tuning on the Mostly Basic Python Problems benchmark and fine-tuning on repaired programs written by humans, improving the pass@1 rate of the Codegen-Mono 6.1B model by 38% relative and 10% absolute.

The paper presents a novel method called Imitation learning from Language Feedback (ILF) to tackle the problem of pretrained language models producing outputs misaligned with human preferences. ILF leverages more informative language feedback through a three-step iterative process: (1) conditioning the language model on input, initial output, and feedback, (2) generating refinements and selecting the one that incorporates the most feedback, and (3) finetuning the language model based on the chosen refinement. Experimental results indicate that ILF effectively scales with dataset size and achieves human-level summarization performance by learning from both language and comparison feedback.

The tuned lens learns an affine transformation to decode the activations of each layer of a transformer as next-token predictions. This provides insights into how model predictions are refined layer by layer. We validate our method on various autoregressive language models up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens baseline.

This article explores the problem of how to train language models to generate text that humans would not consider inappropriate. The authors find that conditional training, which learns the distribution over tokens based on human preference scores, is a simple and effective approach that reduces undesirable content while maintaining downstream task performance. Pre-training LMs with human feedback leads to better preference satisfaction than traditional LM pre-training followed by feedback-based finetuning.

This paper proposes to learn from natural language feedback, which conveys more information per human evaluation than comparisons. It proposes doing so with a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input.

This paper describes a method for improving few-shot learning performance on Natural Language Processing tasks by finetuning on a large number of diverse tasks extracted from internet tables. We find that finetuning on narrow subsets of these tasks can lead to similar improvements, suggesting that the gains are not from domain adaptation but adapting to few-shot learning in general.

This paper discusses the use of reinforcement learning (RL) in fine-tuning large language models to penalize undesirable features in generated sequences. The authors argue that the standard RL approach is flawed and leads to distribution collapse. They propose a Bayesian inference view of KL-regularized RL as a more insightful perspective, which explains how it avoids the distribution collapse problem and offers a first-principles derivation for its objective.