Summary: Reward modeling learns a scalar reward function r(x, y) from human preference data (comparisons, rankings, scores). It is the core component of RLHF — converting human feedback into a differentiable signal for policy optimization. Modern approaches include explicit reward models (Bradley-Terry), implicit rewards (DPO), and multi-objective / constitutional reward models.

The Reward Modeling Problem

Diagram: (Mermaid diagram - view source for diagram code)

flowchart LR
    subgraph HUMAN["Human Feedback Collection"]
        PROMPT[Prompt x] --> POLICY[Policy π(y|x)]
        POLICY --> Y1[Response y₁]
        POLICY --> Y2[Response y₂]
        Y1 --> ANNOTATOR[Human Annotator]
        Y2 --> ANNOTATOR
        ANNOTATOR --> PREF[y₁ ≻ y₂ or y₂ ≻ y₁]
    end

subgraph RM["Reward Model Training"]
    PREF --> DATASET[(Preference Dataset)]
    DATASET --> RM_TRAIN[Train r_φ(x, y)]
end

subgraph RL["Policy Optimization"]
    RM_TRAIN --> PPO[PPO / DPO / etc.]
    PPO --> POLICY
end

classDef human fill:#e3f2fd,stroke:#90caf9;
classDef train fill:#fff3e0,stroke:#ffcc02;
classDef rl fill:#fce4ec,stroke:#f8bbd0;
class PROMPT,POLICY,Y1,Y2,ANNOTATOR,PREF human;
class DATASET,RM_TRAIN train;
class PPO rl;

Preference Modeling: Bradley-Terry-Luce (BTL)

The standard model for pairwise comparisons:

P(y_1 succ y_2 mid x) = (exp(r(x, y_1)))/(exp(r(x, y_1)) + exp(r(x, y_2))) = Sigma(r(x, y_1) - r(x, y_2))

Training objective (negative log-likelihood):

L_R = -E_{(x, y_w, y_l) ~ D} ≤ ft[ log Sigma(r_phi(x, y_w) - r_phi(x, y_l)) right]

Where y_w = preferred (winning), y_l = dispreferred (losing).

Reward Model Architectures

Architecture Description Params Used By
LM Head on Frozen LM Freeze LM, train scalar head on last token ~0.1% of LM InstructGPT, Llama-2-Chat
Full Finetune Finetune entire LM + head 100% GPT-4, Claude (likely)
LoRA on LM Low-rank adapters on frozen LM ~1-5% Open-source RLHF (Zephyr, Tulu)
Separate Smaller Model Distinct encoder (e.g., DeBERTa) 100M-1B Anthropic (early), some open RM

Key Variants & Extensions

1. Multi-Objective / Multi-Attribute Reward Models

r(x, y) = sum_i w_i r_i(x, y)
  • Separate heads for helpfulness, harmlessness, honesty, style, etc.
  • Enables conditional generation (steer at inference via weights)
  • Used in: Constitutional AI, HHH alignment

2. Kahneman-Tversky Optimization (KTO) — Ethayarajh et al., 2024

  • Uses binary feedback (desirable/undesirable) not pairwise
  • Loss: L = E[KT(r(x,y), label)]
  • No need for paired comparisons — scales easier

3. Reward Model Ensembles / Uncertainty

  • Train K RMs with different seeds/data subsets
  • Uncertainty = variance across ensemble
  • Used for: conservative policy updates, active learning, detecting reward hacking

4. Process / Step-Level Reward Models (PRM)

  • Score intermediate reasoning steps, not just final answer
  • Critical for: math, code, multi-step reasoning
  • Training: human-annotated step correctness OR self-consistency
  • Examples: Math-Shepherd, PRM800K, ORA-Math

Reward Hacking & Mitigation

Hacking Mode Description Mitigation
Verbosity Longer responses get higher reward Length penalty, KL constraint
Sycophancy Agreeing with user's false premise Diverse prompts, constitutional AI
Reward Model Exploitation Adversarial inputs fooling RM Ensemble uncertainty, adversarial training
Distribution Shift RM unreliable off-policy KL penalty, early stopping, on-policy data

Evaluation of Reward Models

Metric Description
Accuracy on held-out comparisons % correct on test preference pairs
Correlation with human judgment Spearman/Pearson on Likert/quality scores
Policy win rate vs. baseline End-to-end: does RM-guided policy beat SFT?
Robustness to adversarial attacks Accuracy on perturbed / out-of-distribution inputs
Calibration Does P(y_w succ y_l) match empirical frequency?

Direct Preference Optimization (DPO) — Bypasses Explicit RM

Key insight: The optimal policy under BTL-RM + KL constraint has closed form:

pi^*(y mid x) ∝ pi_{ref}(y mid x) exp((1)/(beta) r^*(x, y))

Substitute into BTL loss → DPO loss (no RM needed):

L_{DPO} = -E_{(x, y_w, y_l)} ≤ ft[ log Sigma ( beta log (pi_theta(y_w mid x))/(pi_{ref)(y_w mid x)} - beta log (pi_theta(y_l mid x))/(pi_{ref)(y_l mid x)} ) right]
Aspect RLHF (PPO + RM) DPO
Training stages 3 (SFT → RM → PPO) 2 (SFT → DPO)
Stability Tricky (PPO hyperparams) Stable (supervised-like)
Compute High (RM + PPO rollouts) Lower (no rollouts)
Quality SOTA (GPT-4, Claude) Competitive (Zephyr, Tulu)
Flexibility RM reusable for many policies Tied to specific policy

For Your Research (PhD Applications)

High-Impact Directions

Direction Why It Matters Feasibility
Uncertainty-aware RM for safe exploration Detect reward hacking, guide active learning Medium (ensemble + GP on embeddings)
Step-level RM for mathematical reasoning Verifiable intermediate steps = better generalization High (PRM800K exists, LEAN integration)
Multi-objective RM with inference-time steering Single model → many behaviors (safety/style/task) Medium (multi-head, conditional)
RM distillation / compression Deploy RM on edge / use as critic in world models High (distill to small encoder)
Constitutional RM without human labels AI feedback (RLAIF) + principles = scalable High (Anthropic recipe)
RM as world model critic Value function for diffusion planning (value-guided) Very High — your world model + robotics focus

Related Wiki Pages

  • rlhf-rl — Full RLHF pipeline
  • dpo — Direct Preference Optimization
  • ppo — Proximal Policy Optimization
  • alignment — AI alignment framework
  • inverse-rl — Learning rewards from behavior
  • constitutional-ai — Principle-based alignment
  • preference-learning — General preference modeling
  • value-guided-diffusion — RM as value function for diffusion planning

Sources

  • instructgpt — Ouyang et al., "Training language models to follow instructions with human feedback" (2022)
  • rlhf-survey-2024 — Comprehensive RLHF survey
  • dpo-paper — Rafailov et al., "Direct Preference Optimization" (2023)
  • kto-paper — Ethayarajh et al., "Kahneman-Tversky Optimization" (2024)
  • prm-paper — Lightman et al., "Let's Verify Step by Step" (2023)