Reinforcement learning from human feedback

In <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>, reinforcement learning from human feedback (RLHF) is a technique to <a href="/facts/AI_alignment/Blgof0uy">align</a> an <a href="/facts/Intelligent_agent/8C2Q93Vw">intelligent agent</a> with human preferences. It involves training a reward model to represent preferences, which can then be used to train other models through <a href="/facts/Reinforcement_learning/NrgPPS0Q">reinforcement learning</a>.
In classical reinforcement learning, an intelligent agent's goal is to learn a function that guides its behavior, called a <a href="/facts/Reinforcement_learning/NrgPPS0Q">policy</a>. This function is iteratively updated to maximize rewards based on the agent's task performance. However, explicitly defining a reward function that accurately approximates human preferences is challenging. Therefore, RLHF seeks to train a "reward model" directly from human <a href="/facts/Feedback/XnuWH3zp">feedback</a>. The reward model is first trained in a <a href="/facts/Supervised_learning/JlNFNPFt">supervised</a> manner to predict if a response to a given prompt is good (high reward) or bad (low reward) based on ranking data collected from human <a href="/facts/Labeled_data/o0doUSPA">annotators</a>. This model then serves as a reward function to improve an agent's policy through an <a href="/facts/Optimization_algorithm/oRn8Iv5I">optimization algorithm</a> like <a href="/facts/Proximal_policy_optimization/7QrmhgRx">proximal policy optimization</a>.

RLHF has applications in various domains in machine learning, including <a href="/facts/Natural_language_processing/1hjMKsSN">natural language processing</a> tasks such as <a href="/facts/Text_summarization/yTQYaEIe">text summarization</a> and <a href="/facts/Conversational_agents/gbMVkBVf">conversational agents</a>, <a href="/facts/Computer_vision/Tl2Yyk66">computer vision</a> tasks like <a href="/facts/Text-to-image_model/ez6APLzp">text-to-image models</a>, and the development of <a href="/facts/Video_game_bot/VJSgUUmr">video game bots</a>. While RLHF is an effective method of training models to act better in accordance with human preferences, it also faces challenges due to the way the human preference data is collected. Though RLHF does not require massive amounts of data to improve performance, sourcing high-quality preference data is still an expensive process. Furthermore, if the data is not carefully collected from a representative <a href="/facts/Sampling_(statistics)/kIb01xdL">sample</a>, the resulting model may exhibit unwanted <a href="/facts/Algorithmic_bias/vHaHsIdV">biases</a>.

Reinforcement learning from human feedback open-in-new

Reinforcement learning from human feedback