1

This article explores the problem of how to train language models to generate text that humans would not consider inappropriate. The authors find that conditional training, which learns the distribution over tokens based on human preference scores, is a simple and effective approach that reduces undesirable content while maintaining downstream task performance. Pre-training LMs with human feedback leads to better preference satisfaction than traditional LM pre-training followed by feedback-based finetuning.

This paper proposes to learn from natural language feedback, which conveys more information per human evaluation than comparisons. It proposes doing so with a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input.

This paper discusses the use of reinforcement learning (RL) in fine-tuning large language models to penalize undesirable features in generated sequences. The authors argue that the standard RL approach is flawed and leads to distribution collapse. They propose a Bayesian inference view of KL-regularized RL as a more insightful perspective, which explains how it avoids the distribution collapse problem and offers a first-principles derivation for its objective.