Summary: Model-Free RL learns a policy or value function directly from environment interactions without learning a dynamics model. Classic algorithms: PPO (on-policy), SAC (off-policy). Higher asymptotic performance but lower sample efficiency than modern MBRL.

Core Algorithms

Algorithm Type Key Idea Best For
PPO (Schulman et al., 2017) On-policy Clipped surrogate objective Stable training, sim
SAC (Haarnoja et al., 2018) Off-policy Max entropy + automatic temp Continuous control
DDPG (Lillicrap et al., 2015) Off-policy Deterministic policy gradient Legacy baseline
TD3 (Fujimoto et al., 2018) Off-policy Twin critics, delayed updates Improved DDPG
PPO-Clip On-policy KL penalty via clipping Standard for LLM RLHF

Model-Free vs Model-Based

Dimension Model-Free (PPO/SAC) Model-Based (Dreamer/TD-MPC)
Samples to convergence 10M-100M+ 1M-5M
Asymptotic performance High (no model bias) Slightly lower (model error)
Planning None (reactive policy) Explicit (MPC/imagination)
Sim-to-real Domain randomization World model adapts
Hyperparameters Domain-specific Fixed (DreamerV3)
Implementation Simple Complex (dynamics + policy)

In LLM / Agent Training

Application Algorithm
RLHF / RLAIF PPO (clip)
Code generation RL PPO + execution reward
Agent loop optimization PPO / GRPO
Tool use training PPO (on-policy) or DPO (offline)

When to Use Which

Scenario Recommendation
Unlimited sim, need max perf PPO/SAC
Limited real data, need efficiency DreamerV3 / TD-MPC
LLM post-training PPO / DPO / GRPO
Dexterous manipulation (real) Diffusion Policy + TD-MPC
Research: understand dynamics MBRL (learns interpretable model)

Related Concepts

Sources

  • Schulman et al., "Proximal Policy Optimization" (2017)
  • Haarnoja et al., "Soft Actor-Critic" (2018)
  • model-based-rl — Comparison table