Publications

After testing 23 LLMs from 8 model series, we find that larger LLMs are more vulnerable to poisoning. Our jailbreak-tuning method causes GPT-4o to capably answer virtually any harmful question. We investigated the vulnerability to three forms of data poisoning: malicious fine-tuning, imperfect data curation, and intentional data contamination.

While larger language models exhibit impressive capabilities, they remain vulnerable to adversarial prompts. Empirical findings show that robustness against such attacks significantly improves with adversarial training, but not with model scaling alone.

To understand how neural networks generalize, we studied an RNN trained to play Sokoban. The RNN learned to plan ahead by ‘pacing’ despite penalties, demonstrating that reinforcement learning can encourage strategic planning in neural networks.

Evaluating three neural network circuits (IOI, greater-than, and docstring) under adversarial conditions reveals that the IOI and docstring circuits fail to match the full model’s behavior even on benign inputs, underscoring the need for more robust circuits in safety-critical applications.

Reinforcement learning from human feedback (RLHF) uses KL divergence regularization to mitigate reward errors, allowing high utility with light-tailed errors but suffering from reward hacking with heavy-tailed errors. While current models have light-tailed errors, real-world applications may still face significant risks from heavy-tailed errors, leading to catastrophic Goodhart.

InterpBench is a collection of 17 semi-synthetic transformers with known circuits, trained using Strict Interchange Intervention Training (SIIT). These models exhibit realistic weights and activations that reflect ground truth circuits, providing a valuable benchmark for evaluating mechanistic interpretability techniques.

By adapting existing interpretability techniques to the Mamba architecture, we partially reverse-engineered the circuit responsible for the Indirect Object Identification task, identifying layer 39 as a key component, and demonstrating the potential of these techniques to generalize to new architectures.

Existing circuits in the mechanistic interpretability literature may not be as faithful as reported. Current circuit faithfulness scores reflect both the methodological choices of researchers and the actual components of the circuit.

Guaranteed safe (GS) AI ensures AI systems avoid harmful behaviors through a world model, safety specification, and verifier to provide high-assurance safety guarantees. The paper discusses creating these components, technical challenges, and argues for the necessity and superiority of this approach to AI safety.