RL for Frontier LLMs — Curriculum

A progressive curriculum: solid RL foundations first, then the bridge to language models, then current SOTA for training frontier LLMs. Each lesson is a self-contained Obsidian learning guide (dependency map → motivated teaching → quizzes → practice → reading order).

How to work through this

Lessons are ordered by dependency. Don’t skip ahead — each tier assumes the previous one. Foundation lessons are stable classical RL; the SOTA tier is verified against current sources and will evolve.

Progress tracker

How to track progress

Each lesson has a status property (open a lesson, edit the status field in its Properties panel): unread → reading → done, or review to revisit. The table below is a live Dataview query — it updates automatically as you change statuses.

TABLE WITHOUT ID
  file.link AS "Lesson",
  choice(status = "done", "✅ done",
    choice(status = "reading", "📖 reading",
      choice(status = "review", "🔁 review", "⬜ unread"))) AS "Status"
FROM "wiki/learn/rl"
SORT file.name ASC

Overall:

TABLE WITHOUT ID
  length(filter(rows.status, (s) => s = "done")) AS "✅ Done",
  length(filter(rows.status, (s) => s = "reading")) AS "📖 Reading",
  length(filter(rows.status, (s) => s = "unread")) AS "⬜ Unread",
  length(rows) AS "Total"
FROM "wiki/learn/rl"
GROUP BY true

Dependency map

graph TD
    subgraph "Tier 1 — Foundations"
        L1["01 · MDPs & the RL objective"]
        L2["02 · Policy gradients<br/>REINFORCE, PG theorem, baselines"]
        L3["03 · Actor-critic & GAE<br/>advantage estimation"]
        L4["04 · PPO<br/>TRPO → clipped objective"]
    end
    subgraph "Tier 2 — Bridge to language"
        L5["05 · RL on token sequences<br/>the LLM MDP, KL-to-reference"]
        L6["06 · RLHF pipeline<br/>reward modeling, InstructGPT"]
    end
    subgraph "Tier 3 — SOTA"
        L7["07 · DPO & RL-free preference opt<br/>IPO, KTO, ORPO"]
        L8["08 · Scaling RLHF & alternatives<br/>RLAIF, Constitutional AI, reward hacking"]
        L9["09 · RL for reasoning<br/>RLVR, GRPO, PRM vs ORM, R1-style"]
        L10["10 · Frontier & infra<br/>async rollouts, overoptimization, 2025-26 SOTA"]
    end

    L1 --> L2 --> L3 --> L4
    L4 --> L5 --> L6
    L6 --> L7
    L6 --> L8
    L6 --> L9
    L7 --> L9
    L8 --> L10
    L9 --> L10

    style L1 fill:#44a,color:#fff
    style L2 fill:#44a,color:#fff
    style L3 fill:#44a,color:#fff
    style L4 fill:#44a,color:#fff
    style L9 fill:#4a4,color:#fff
    style L10 fill:#4a4,color:#fff

Lessons

Tier 1 — Foundations (classical RL, stable) ✅

• 01-mdps-and-the-rl-objective — MDPs, returns, value/advantage functions, the RL objective
• 02-policy-gradients — REINFORCE, the policy gradient theorem, baselines & variance reduction
• 03-actor-critic-and-gae — critics, the bias-variance spectrum, GAE
• 04-ppo — trust regions → the clipped surrogate; the RLHF workhorse

Tier 2 — Bridge to language models ✅

• 05-rl-on-token-sequences — the LLM as a token-level MDP; per-token KL-to-reference; PPO on tokens
• 06-rlhf-pipeline — SFT → Bradley-Terry reward model → PPO; reward overoptimization

Tier 3 — SOTA for frontier LLMs ✅

• 07-dpo-and-rl-free-preference-optimization — DPO derivation; IPO/KTO/ORPO/SimPO; the RL-free family
• 08-scaling-rlhf-and-alternatives — RLAIF, Constitutional AI, reward overoptimization, Best-of-N/RFT/ReST
• 09-rl-for-reasoning — RLVR, GRPO, R1-style training, PRM vs ORM, the 2025–26 landscape
• 10-frontier-and-infra — async RL systems, agentic/multi-turn RL, entropy collapse, open problems

Design principle

Every lesson — even the foundational ones — ends with an “LLM connection” section tying the concept back to how frontier models are actually trained today. The goal is not RL-in-general but RL-as-practiced-for-LLMs.