The Vienna Alignment Workshop gathered researchers to explore critical AI safety issues, including Robustness, Interpretability, Guaranteed Safe AI, and Governance, with a keynote by Jan Leike. It was followed by an informal Unconference, fostering further discussions and networking.

Giving RNNs extra thinking time at the start boosts their planning skills in Sokoban. We explore how this planning ability develops during reinforcement learning. Intriguingly, we find that on harder levels the agent paces around to get enough computation to find a solution.

Frontier LLMs like ChatGPT are powerful but not always robust. Scale helps with many things. We wanted to see if scaling up the model size can ‘solve’ robustness issues.

Achieving robustness remains a significant challenge even in narrow domains like Go. We test three approaches to defend Go AIs from adversarial strategies. We find these defenses protect against previously discovered adversaries, but uncover qualitatively new adversaries that undermine these defenses.

We conducted 17 semi-structured interviews of AI safety experts about their big picture strategic view of the AI safety landscape: how will human-level AI play out, how things might go wrong, and what should the AI safety community be doing. While many respondents held “traditional” views (e.g. the main threat is misaligned AI takeover), there was more opposition to these standard views than we expected, and the field seems more split on many important questions than someone outside the field may infer.

We present LLMs with a series of moral choices and find that LLMs tend to align with human judgement in clear scenarios. In ambiguous scenarios most models exhibit uncertainty, but a few large proprietary models share a set of clear preferences.

The New Orleans Alignment Workshop 2023 brought together leading ML researchers working to align advanced AI with human values and develop safe AI system. Presentations from industry, academia, and non-profits focused on topics spanning from oversight, interpretability, robustness, generalization and governance.

We red-team three new functionalities exposed in the GPT-4 APIs: fine-tuning, function calling and knowledge retrieval. We find that fine-tuning a model on as few as 15 harmful examples or 100 benign examples can remove core safeguards from GPT-4, enabling a range of harmful outputs. Furthermore, we find that GPT-4 Assistants readily divulge the function call schema and can be made to execute arbitrary function calls. Finally, we find that knowledge retrieval can be hijacked by injecting instructions into retrieval documents.

End-of-year round up of FAR.AI’s activities. Our research has culminated in 13 academic papers across robustness, value alignment and model evaluations; our field building events have reached more than 160 ML experts; and our coworking space hosts 40 members working on AI safety.

Highlights from FAR.AI’s alignment research in 2023. Our science of robustness agenda has found vulnerabilities in superhuman Go systems; our value alignment research has developed more sample-efficient value learning algorithms; and our model evaluation direction has developed a variety of new black-box and white-box evaluation methods.

Prominent AI scientists from China and the West propose joint strategy to mitigate risks from AI at the inaugural International Dialogue on AI Safety.

We demonstrate Codebook Features: a way to modify neural networks to make their internals more interpretable and steerable while causing only a small degradation of performance. At each layer, we apply a quantization bottleneck that forces the activation vector into a sum of a few discrete codes; converting an inscrutable, dense, and continuous vector into a discrete list of codes from a learned ‘codebook’ that are either on or off.

We show how to use Vision-Language Models (VLM), and specifically CLIP models, as reward models (RM) for RL agents. Instead of manually specifying a reward function, we only need to provide text prompts like ‘a humanoid robot kneeling’ to instruct and provide feedback to the agent. Importantly, we find that larger VLMs provide more accurate reward signals, so we expect this method to work even better with future models.

A one-dimensional PCA projection of OpenAI’s text-embedding-ada-002 model achieves 73.7% accuracy on the ETHICS Util test dataset. This is comparable with the 74.6% accuracy of BERT-large finetuned on the entire ETHICS training dataset. This demonstrates language models develop implicit representations of human utility purely from self-supervised learning.

Our adversarial testing algorithm uncovers a simple, human-interpretable strategy that consistently beats superhuman Go AIs. We explore the implications this has for the robustness and safety of AI systems.

All contemporary machine learning systems are vulnerable to adversarial attack. This poses serious problems for existing alignment proposals. We explore these issues and propose several research directions FAR.AI is pursuing to overcome this challenge.