Summary: RLHF (Reinforcement Learning from Human Feedback) aligns LLMs with human preferences through a three-stage pipeline: Supervised Fine-Tuning (SFT) → Reward Modeling (RM) → Policy Optimization (PPO/DPO). Modern variants include DPO (direct optimization), KTO (binary feedback), RLAIF (AI feedback), and online/iterative RLHF. This is the dominant paradigm for instruction-following, safety, and capability alignment.

The RLHF Pipeline

Diagram: (Mermaid diagram - view source for diagram code)

flowchart TD
    subgraph STAGE1["Stage 1: Supervised Fine-Tuning (SFT)"]
        BASE[Base LLM] --> SFT_DATA[High-quality\nInstruction-Response Pairs]
        SFT_DATA --> SFT[SFT: π_SFT]
    end

subgraph STAGE2["Stage 2: Reward Modeling"]
    SFT --> POLICY_SFT[Generate\nResponses]
    POLICY_SFT --> Y1[y₁]
    POLICY_SFT --> Y2[y₂]
    Y1 --> HUMAN[Human\nAnnotators]
    Y2 --> HUMAN
    HUMAN --> PREF[y₁ ≻ y₂]
    PREF --> RM_TRAIN[Train r_φ(x,y)\nBradley-Terry]
end

subgraph STAGE3["Stage 3: Policy Optimization"]
    RM_TRAIN --> PPO[PPO / DPO / KTO]
    PPO --> POLICY_RL[Aligned Policy π_RL]
end

classDef sft fill:#e8f5e9,stroke:#c8e6c9;
classDef rm fill:#fff3e0,stroke:#ffcc02;
classDef rl fill:#fce4ec,stroke:#f8bbd0;
class BASE,SFT_DATA,SFT,POLICY_SFT,Y1,Y2,HUMAN,PREF sft;
class RM_TRAIN rm;
class PPO,POLICY_RL rl;

Stage 1: Supervised Fine-Tuning (SFT)

Aspect Details
Data 10K-100K high-quality (prompt, response) pairs
Sources Human-written, distilled from stronger models, synthetic
Objective L_{SFT} = -E[log pi_theta(y mid x)]
Result π_SFT: follows format, basic instruction following
Key Insight SFT teaches format and style; RL teaches preference and quality

Stage 2: Reward Modeling

See reward-modeling for full details.

Core: Learn r_phi(x, y) from human comparisons using Bradley-Terry model.

Output: Scalar reward for any (prompt, response) pair.

Stage 3: Policy Optimization

PPO (Proximal Policy Optimization) — Original RLHF

Diagram: (Mermaid diagram - view source for diagram code)

flowchart LR
    PROMPT[x] --> POLICY[π_θ]
    POLICY --> RESP[y]
    RESP --> REWARD[r_φ(x,y)]
    RESP --> REF[π_ref = π_SFT]
    REWARD --> OBJ[PPO Objective]
    REF --> KL[KL Penalty]
    KL --> OBJ
    OBJ --> UPDATE[θ ← θ + ∇L_PPO]

PPO Objective (with KL penalty):

L_{PPO} = E_{x ~ D, y ~ pi_theta} ≤ ft[ r_phi(x, y) - beta log (pi_theta(y mid x))/(pi_{ref)(y mid x)} right] + PPO-clip term

Key hyperparameters: beta (KL coeff), clip epsilon, learning rate, minibatch size, epochs.

DPO (Direct Preference Optimization) — Rafailov et al., 2023

Key insight: Optimal policy under BTL-RM + KL constraint has closed form:

pi^*(y mid x) ∝ pi_{ref}(y mid x) exp((1)/(beta) r^*(x, y))

Substitute into BTL loss → no RM needed:

L_{DPO} = -E_{(x, y_w, y_l)} ≤ ft[ log Sigma ( beta log (pi_theta(y_w mid x))/(pi_{ref)(y_w mid x)} - beta log (pi_theta(y_l mid x))/(pi_{ref)(y_l mid x)} ) right]
Aspect PPO + RM DPO
Stages 3 (SFT → RM → PPO) 2 (SFT → DPO)
Stability Sensitive to PPO hyperparams Stable (supervised-like)
Compute High (RM + rollouts) Lower (no rollouts)
Quality SOTA (GPT-4, Claude) Competitive (Zephyr, Tulu)
RM Reusability Yes (RM → many policies) No (policy-specific)

KTO (Kahneman-Tversky Optimization) — Ethayarajh et al., 2024

  • Feedback: Binary (desirable / undesirable), not pairwise
  • Loss: Prospect theory utility function
  • Advantage: Easier data collection, no paired comparisons needed

RLHF Variants Taxonomy

Diagram: (Mermaid diagram - view source for diagram code)

graph TD
    RLHF[RLHF Variants] --> ONLINE[Online / Iterative RLHF]
    RLHF --> OFFLINE[Offline RLHF]
    RLHF --> MODEL_FREE[Model-Free]
    RLHF --> MODEL_BASED[Model-Based]

ONLINE --> ITER1[Iterative DPO / IPO]
ONLINE --> ITER2[Self-Play / Self-Rewarding]
ONLINE --> ITER3[Online PPO with Fresh RM]

OFFLINE --> DPO1[DPO / IPO / KTO]
OFFLINE --> DPO2[Offline PPO]

MODEL_FREE --> MF1[PPO / DPO / KTO]
MODEL_BASED --> MB1[World Model + Planning]
MODEL_BASED --> MB2[Value-Guided Diffusion]

Evaluation Benchmarks

Benchmark Focus Metric
MT-Bench Multi-turn conversation GPT-4 judge score
AlpacaEval 2.0 Instruction following Win rate vs. GPT-4
Arena-Hard Challenging prompts Human preference
TruthfulQA Truthfulness / hallucination Accuracy
HarmBench Safety / refusal Attack success rate
MMLU / GSM8K Capability retention Accuracy

Common Failure Modes & Fixes

Failure Symptom Fix
Reward Hacking High RM score, poor human eval Ensemble RM, KL penalty, adversarial training
Mode Collapse Low diversity, repetitive outputs Higher temp, entropy bonus, diverse prompts
Catastrophic Forgetting Loss of base capabilities Stronger KL, replay buffer, multi-task SFT
Sycophancy Agrees with false user premises Constitutional AI, diverse training prompts
Over-Optimization RM score ↑, human eval ↓ Early stopping, conservative β, offline eval

For Your Research (PhD Applications)

High-Impact Directions Aligned With Your Interests

Direction Connection to Your Profile Feasibility
RLHF for World Models / Diffusion Policies Your robotics + world model focus Very High — novel, high impact
Value-Guided Diffusion = RLHF in Latent Space value-guided-diffusion, world-model Very High — your core theme
Step-Level RM for Planning Verification Math reasoning + planning High — PRM800K + your HRM interest
Online/Iterative RLHF with World Model Rollouts Model-based RL + alignment Medium-High — needs compute
Multi-Objective RM for Embodied Agents Safety + task + efficiency tradeoffs High — robotics needs this
RLHF without Human Labels (RLAIF + Constitutional) Scalable alignment High — Anthropic recipe works

Implementation Resources

# DPO (simplest to implement)
from trl import DPOTrainer

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    beta=0.1,
    train_dataset=dataset,  # (prompt, chosen, rejected)
    tokenizer=tokenizer,
)
trainer.train()

# PPO (more complex)
from trl import PPOTrainer, AutoModelForCausalLMWithValueHead

ppo_model = AutoModelForCausalLMWithValueHead.from_pretrained(sft_model)
trainer = PPOTrainer(
    model=ppo_model,
    reward_model=reward_model,
    tokenizer=tokenizer,
    kl_coef=0.05,
)
# Rollout → reward → PPO step loop

Related Wiki Pages

  • reward-modeling — Core RM theory & variants
  • dpo — Direct Preference Optimization deep-dive
  • ppo — PPO algorithm details
  • alignment — Broader alignment framework
  • sft — Supervised Fine-Tuning
  • constitutional-ai — Principle-based alignment
  • rlaif — AI feedback (Constitutional AI)
  • kto — Binary feedback optimization
  • value-guided-diffusion — RLHF-style value guidance for diffusion
  • world-model — World models as policies/critics

Sources

  • instructgpt — Ouyang et al., "Training language models to follow instructions with human feedback" (2022)
  • dpo-paper — Rafailov et al., "Direct Preference Optimization" (2023)
  • kto-paper — Ethayarajh et al., "Kahneman-Tversky Optimization" (2024)
  • rlhf-survey-2024 — Comprehensive RLHF survey
  • constitutional-ai — Bai et al., "Constitutional AI" (2022)
  • rlaif — Lee et al., "RLAIF: Scaling RLHF with AI Feedback" (2023)