1

This paper proposes to learn from natural language feedback, which conveys more information per human evaluation than comparisons. It proposes doing so with a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input.

This paper discusses the use of reinforcement learning (RL) in fine-tuning large language models to penalize undesirable features in generated sequences. The authors argue that the standard RL approach is flawed and leads to distribution collapse. They propose a Bayesian inference view of KL-regularized RL as a more insightful perspective, which explains how it avoids the distribution collapse problem and offers a first-principles derivation for its objective.