10 · Frontier RL — Infra & Open Problems

What you're learning

Why RL post-training at frontier scale is, first and foremost, a systems problem — and where the science is still open. By the end you should be able to (1) explain why generation, not the gradient step, dominates wall-clock and dictates the whole architecture; (2) reason about disaggregated, asynchronous, off-policy training and the staleness corrections that make it sound; (3) survey the open-source framework landscape and what each is for; and (4) speak precisely about the field’s live debates — entropy collapse, credit assignment over long CoT, reward beyond verifiable domains, and whether RL creates capability or merely elicits it.

This is the capstone (Tier 3, lesson 10). It assumes the whole curriculum: PPO (04-ppo), the token-level MDP and KL-to-reference (05-rl-on-token-sequences), the RLHF pipeline and reward overoptimization (06-rlhf-pipeline), and especially RL for reasoning / RLVR / GRPO (09-rl-for-reasoning). Where earlier lessons asked “what is the objective and why,” this one asks “how do you actually run it on thousands of GPUs, and what don’t we know yet?”

This is a landscape lesson — read the confidence labels

The frontier moves monthly. Throughout, I mark claims as [established] (documented in papers/repos), [reported] (credibly inferred from tech reports or strong secondary sources), or [open/contested] (an active debate). Framework features and lab recipes are verified against sources as of September 2026 and will drift. Trust the mechanism; re-verify the specifics.

---

1. Learning map

graph TD
    A["RLVR/GRPO objective<br/>(lesson 09)"] --> B["Observation:<br/>a training step = generate + score + update"]
    B --> C["Generation dominates<br/>wall-clock (60–90%)"]
    C --> D["→ RL post-training is<br/>a systems problem"]
    D --> E["Generation ≠ training:<br/>different engines, different HW profile"]
    E --> F["Actor–learner disaggregation<br/>(IMPALA lineage)"]
    F --> G["Colocated vs disaggregated<br/>placement"]
    F --> H["Async / off-policy rollouts<br/>→ staleness"]
    H --> I["Importance-sampling<br/>corrections (V-trace, TIS,<br/>decoupled PPO)"]
    H --> J["Replay / prompt buffers"]
    E --> K["KV-cache reuse<br/>+ verifier/sandbox infra"]
    G --> L["Frameworks:<br/>verl · OpenRLHF · TRL ·<br/>NeMo-RL · AReaL · slime · SkyRL"]
    I --> L
    K --> L
    L --> M["What frontier labs do<br/>(Sept 2026 synthesis)"]
    C --> N["On-policy distillation &<br/>RL-distill hybrids"]
    A --> O["Open problems"]
    O --> O1["Entropy collapse<br/>& exploration"]
    O --> O2["Credit assignment<br/>over long CoT"]
    O --> O3["Reward beyond<br/>verifiable domains"]
    O --> O4["RLVR: elicit or<br/>create capability?"]
    O --> O5["Scaling laws for<br/>RL post-training"]
    O1 --> M
    O4 --> M

    style D fill:#a44,color:#fff
    style L fill:#44a,color:#fff
    style M fill:#4a4,color:#fff
    style O fill:#a64,color:#fff

Prerequisites (assumed): GRPO and RLVR, verifiers vs learned reward models, the KL-to-reference leash and reward overoptimization, PPO’s clipped surrogate and importance-sampling ratio. If any is shaky, revisit 09-rl-for-reasoning and 06-rlhf-pipeline first.

---

2. Why this matters: the bottleneck moved

Through lesson 09 we treated an RL step as an equation. At frontier scale the equation is the easy part. A single RLVR step is really a pipeline:

$\underset{\text{1. generate (autoregressive decode)}}{\underbrace{y\sim {\pi }_{\theta }(\cdot \mid x)}}⟶\underset{\text{2. score (sandbox / checker)}}{\underbrace{r=\text{verify}(x,y)}}⟶\underset{\text{3. update (one backward pass)}}{\underbrace{\theta \leftarrow \theta +\eta \hat{g}}}$

Here is the fact that reorganizes everything below: step 1 dominates wall-clock, often by an order of magnitude over step 3. In reasoning RL you sample long chains of thought — thousands of tokens per rollout, a group of $G$ samples per prompt for GRPO, hundreds to thousands of prompts per step. Generation is memory-bandwidth-bound autoregressive decoding; the update is one compute-bound forward+backward pass over already-generated tokens. They are not the same workload, and they do not want the same machine.

Concrete, verified numbers [established]:

• veRL’s reproduction of DAPO on Qwen-32B spends roughly ~70% of step time in rollout generation (veRL / HybridFlow profiling; the exact fraction rises with response length and group size).¹
• NVIDIA’s NeMo-RL reports generation at ~65–72% of step time for long-CoT reasoning workloads.²
• HuggingFace’s TRL team frames the entire async-rollout effort (“Keep the Tokens Flowing”) around the same observation: the learner starves while generators decode.³

The one sentence to remember

RL post-training is a distributed-systems problem wearing an optimization problem’s clothes. The algorithm (GRPO/PPO) is settled enough that throughput — how fast you turn GPUs into fresh, correctly-scored rollouts — is what actually gates progress. Every architectural choice in this lesson follows from “make generation not be the bottleneck.”

Quiz 1: Why does generation dominate the cost of an RL post-training step, and why can't you just "use a bigger batch" to fix it?

Answer

Why generation dominates. A rollout is autoregressive decoding: to produce a $T$-token response you run $T$ sequential forward passes, each depending on the last, each reading the full KV cache from HBM. This is memory-bandwidth-bound and latency-bound — you cannot parallelize across the token dimension. In reasoning RL, $T$ is huge (thousands of CoT tokens), you draw a group of $G$ samples per prompt (GRPO), and you do it for a large prompt batch — so you pay $O(\text{prompts}\times G\times T)$ sequential decode steps. The gradient update, by contrast, is a single forward+backward over the already-materialized tokens: compute-bound, fully parallel across the sequence, done in one shot. Measured fractions land at ~65–90% generation for long-CoT workloads (veRL, NeMo-RL, TRL).

Why bigger batches don’t fix it. Enlarging the batch raises throughput (tokens/sec) but not the latency of a step, and it does nothing about the core imbalance: while generators decode, the learner GPUs sit idle, and vice versa (in a naïve synchronous, colocated design). Bigger batches also make the tail worse — a step must wait for its slowest, longest rollout to finish before the update can run (straggler / long-tail problem). The real fixes are structural: disaggregate generation from training onto engines tuned for each, and run them asynchronously so neither waits on the other (§4–§5). Batch size is a knob; the bottleneck is a pipeline-topology problem.

---

3. Anatomy of a disaggregated, asynchronous RL system

Before the details, here is the target architecture the whole field is converging on — a disaggregated actor–learner system with a decoupled verifier and a queue/buffer between generation and training.

flowchart LR
    subgraph GEN["Generators / rollout workers (inference engines)"]
        G1["vLLM / SGLang replica<br/>weights θ_behavior<br/>KV-cache + prefix reuse"]
        G2["vLLM / SGLang replica"]
        G3["... N replicas ..."]
    end

    subgraph VER["Verifier / environment layer"]
        V1["Sandboxed code exec<br/>(isolated containers)"]
        V2["Math checker / answer match"]
        V3["Tool APIs · agent env<br/>(multi-turn)"]
    end

    subgraph BUF["Replay / rollout queue"]
        Q["(prompt buffer +<br/>completed, scored rollouts;<br/>staleness tag: age in updates)"]
    end

    subgraph LRN["Learner (training engine)"]
        L1["FSDP / Megatron<br/>policy π_θ (+ value head)<br/>grad update"]
        L2["staleness / IS<br/>correction"]
    end

    PROMPTS["Prompt dataset"] --> Q
    Q -->|"assign prompts"| GEN
    GEN -->|"completions y"| VER
    VER -->|"reward r"| Q
    Q -->|"sample batch<br/>(bounded staleness)"| LRN
    LRN -->|"weight sync<br/>(NCCL / RDMA broadcast,<br/>every k steps)"| GEN

    style GEN fill:#223,color:#fff
    style LRN fill:#322,color:#fff
    style VER fill:#232,color:#fff
    style BUF fill:#332,color:#fff

Read the loop: generators hold a (possibly slightly stale) copy of the policy ${\theta }_{\text{behavior}}$ and decode completions using an inference engine (vLLM/SGLang) with KV-cache and prefix reuse; the verifier layer scores them (code sandbox, math checker, tool/agent environment); scored rollouts land in a queue/buffer tagged with their staleness (how many learner updates old the generating weights are); the learner pulls bounded-staleness batches, applies an importance-sampling / staleness correction, takes a gradient step, and periodically broadcasts fresh weights back to the generators. Nothing blocks: generators keep decoding while the learner trains.

This is not new in spirit. It is the actor–learner decomposition of IMPALA (Espeholt et al., 2018, arXiv:1802.01561) — many actors generate experience asynchronously, a central learner consumes it — adapted to the LLM regime where “an actor” is a vLLM replica and “experience” is a scored CoT. IMPALA also bequeathed the correction we’ll need in §5: V-trace.⁴

The three-way hardware split

These three roles have different optimal hardware/parallelism. Generation wants throughput-optimized inference (tensor parallelism, paged KV-cache, continuous batching — vLLM/SGLang). Training wants FSDP/Megatron sharding tuned for backward passes. Verification wants CPU-heavy, horizontally-scaled, isolated sandboxes. Trying to serve all three from one code path and one placement is exactly what the frameworks in §7 exist to avoid — or to deliberately colocate and time-share, which is the central design axis (§4).

---

4. The generation-vs-training split: colocated vs disaggregated

Given two very different workloads (generate, train), you have two placement strategies. This is the architectural fork.

Colocated (time-sharing). Put the inference engine and the trainer on the same GPUs and alternate: generate → free the KV cache → reshard weights → train → reshard back → repeat. Memory is reused, so you can devote all GPUs to whichever phase is active. veRL’s HybridFlow (Sheng et al., arXiv:2409.19256, EuroSys 2025) is the canonical example; its 3D-HybridEngine does zero-redundancy resharding of weights between the training layout (e.g. FSDP/Megatron) and the rollout layout (vLLM TP) with minimal communication.¹ Downside: the phases are still synchronous — the learner is idle during generation and vice versa — and you pay a resharding/transition cost each switch.

Disaggregated (dedicated pools). Give generation its own GPU pool and training its own, connected by the queue in §3. Now they run concurrently and can be asynchronous (§5). OpenRLHF (Hu et al., arXiv:2405.11143) pioneered this style for RLHF: Ray places vLLM generators, DeepSpeed-ZeRO trainers, and RM/reference models as separate schedulable actors.⁵ Downside: you must size the two pools to balance throughput (too few generators → learner starves; too few learners → rollouts pile up stale), and you must move weights across the network.

"Colocated vs disaggregated" is not "synchronous vs async"

They’re orthogonal-ish but correlated. You can run colocated-synchronous (classic verl), disaggregated-synchronous (OpenRLHF’s default), or disaggregated-asynchronous (AReaL, async TRL, Llama-4-style). Colocated-async is awkward because time-sharing the same GPUs fights with running both at once. The industry drift in 2025–26 is toward disaggregated + async for large reasoning runs, because that’s what keeps both GPU pools saturated. Small/medium runs still often use colocated verl for simplicity and memory efficiency.

The programming-model wrinkle: single- vs multi-controller

HybridFlow’s other contribution is a hybrid controller model. A single-controller (one driver orchestrating everything, à la classic distributed RL) is easy to write but bottlenecks on the driver at scale; a multi-controller (SPMD, each worker runs the same program) scales but is painful to express complex dataflow in. verl uses a single-controller for the inter-role dataflow (the RL algorithm’s high-level “generate → score → update” graph) and multi-controller within each role (the actual sharded compute). This is why verl code reads like a clean RL loop while still scaling — a genuinely influential design that later frameworks borrowed [established].¹

---

5. Async and off-policy at scale: staleness and its corrections

Asynchrony buys throughput but breaks a assumption baked into PPO/GRPO: on-policyness. If generators run ahead while the learner updates, the rollouts the learner consumes were produced by an older policy ${\pi }_{{\theta }_{\text{old}}}$ (the behavior policy) than the current ${\pi }_{\theta }$ (the target policy). The data is off-policy, by however many updates of “staleness” you allow.

5.1 Why a little off-policyness is fine — and how much is too much

[established] Noukhovitch et al., “Asynchronous RLHF” (arXiv:2410.18252, ICLR 2025), showed that a modest generation-training lag is nearly free in quality while giving large speedups — you can decouple generation from training and tolerate the resulting off-policy gap if you bound it.⁶ AReaL (Fu et al., arXiv:2505.24298, NeurIPS 2025) makes the knob explicit: a staleness bound $\eta$ (rollouts may be at most $\eta$ updates old), reporting strong results up to $\eta \approx 4$–$8$ and a ~2.77× throughput win over synchronous, at matched final accuracy — with interruptible / partial-rollout generation (pause a long decode, sync weights, resume) so no single long CoT stalls the whole step.⁷

The frontier regime is "near-on-policy," not "replay-buffer off-policy"

This is a crucial calibration. Frontier reasoning RL is not DQN-style off-policy learning from a giant replay buffer of ancient experience. It is near-on-policy: a few updates of staleness, tightly bounded, with a correction to patch the small residual bias. Deep replay buffers largely do not help LLM RL (the policy moves too fast; old rollouts become useless and even harmful). Recent replay work (RLEP, arXiv:2507.07451⁸; RePO, arXiv:2506.09340⁹) mostly replays high-value successful rollouts to fight sparsity, not to enable deep off-policyness. Treat “off-policy at scale” as “a controlled few-step lag,” not “learn from anything, anytime.” [established, with caveats]

5.2 The correction: importance sampling and its clipped/truncated variants

When behavior $\mu ={\pi }_{{\theta }_{\text{old}}}$ differs from target $\pi ={\pi }_{\theta }$, an off-policy estimate of the policy gradient reweights each sample by the importance ratio $\rho =\pi (y\mid x)/\mu (y\mid x)$. Two lineages matter:

V-trace (IMPALA). Corrects an off-policy value/advantage target with truncated IS weights. Define per-token ${\rho }_t=\pi (a_t\mid s_t)/\mu (a_t\mid s_t)$; V-trace uses clipped weights ${\bar{\rho }}_t=\min (\bar{\rho },{\rho }_t)$ and ${\bar{c}}_t=\min (\bar{c},{\rho }_t)$ so that a few large ratios can’t blow up the estimate. The truncation trades a little bias for a lot of variance reduction — the recurring theme of every correction here.

Decoupled PPO / clipped IS for LLMs. PPO already contains an IS ratio and a clip (lesson 04). The async twist: there are now two distributions in play — the generation policy (what actually sampled the tokens) and the proximal ${\theta }_{\text{old}}$ of the PPO surrogate. AReaL’s decoupled PPO objective separates them, applying an explicit IS correction from the (stale) behavior policy to the current policy on top of the usual clip. This is what makes bounded-staleness training sound rather than just lucky.

Your "on-policy" RL framework is probably secretly off-policy

A subtle, load-bearing 2025 finding [established, active]: even in a synchronous setup, the tokens are generated by an inference engine (vLLM/SGLang) while the log-probs used in the loss are recomputed by the training engine (FSDP/Megatron). These use different kernels, different precision, different attention implementations — so ${\pi }_{\text{infer}}(y\mid x)\ne {\pi }_{\text{train}}(y\mid x)$ for the same weights. That gap is a hidden off-policy bias that can silently degrade or destabilize training. The community fix is Truncated Importance Sampling (TIS) — reweight by ${\pi }_{\text{train}}/{\pi }_{\text{infer}}$ with a truncation cap — to reconcile the two engines (see the “rollout–training mismatch” line of work, e.g. Sea AI Lab’s analyses and the FP16-precision remedy, arXiv:2510.26788).¹⁰ Moral: importance sampling isn’t optional bookkeeping for async — it’s needed even when you think you’re on-policy, because two engines are never bitwise the same distribution.

${\hat{g}}_{\text{corrected}}={\mathbb{E}}_{y\sim \mu }[\underset{\text{truncated IS}}{\underbrace{\min (\rho ,\bar{\rho })}}\hat{A}(x,y){\nabla }_{\theta }\log {\pi }_{\theta }(y\mid x)],\rho =\frac{{\pi }_{\theta }(y\mid x)}{\mu (y\mid x)}.$

The whole art is choosing $\bar{\rho }$ (and whether $\rho$ is token- or sequence-level): too tight and you throw away signal, too loose and a handful of high-ratio samples wreck the update. Sequence-level ratios (as in GSPO, arXiv:2507.18071) are lower-variance but coarser; token-level (GRPO) is finer but noisier — an active design axis, not a solved question [open].¹¹

Quiz 2: In an asynchronous actor–learner setup, the learner receives rollouts generated by a policy a few updates stale. (a) Why is a naïve PPO/GRPO update on this data biased? (b) What correction fixes it, and what's the bias–variance knob? (c) Why might you still need this correction in a fully synchronous run?

Answer

(a) Why biased. PPO/GRPO estimate ${\nabla }_{\theta }J$ as an expectation over samples from the current policy ${\pi }_{\theta }$. In async training the samples come from a stale behavior policy $\mu ={\pi }_{{\theta }_{\text{old}}}$. Optimizing ${\mathbb{E}}_{y\sim \mu }[\dots ]$ as if $y\sim {\pi }_{\theta }$ is a distribution-mismatch bias: you’re crediting/blaming the current policy for actions an older policy chose, and the advantage estimates were computed under the old policy’s state distribution.

(b) The fix. Reweight by the importance ratio $\rho ={\pi }_{\theta }(y\mid x)/\mu (y\mid x)$ so the off-policy samples become an unbiased(-ish) estimator for the current policy — the V-trace / decoupled-PPO idea. Raw $\rho$ is high-variance (ratios can explode when the policies diverge), so you truncate/clip it ($\min (\rho ,\bar{\rho })$). The cap $\bar{\rho }$ is the bias–variance knob: tighter cap → lower variance, more bias (you underweight genuinely-changed regions); looser cap → less bias, more variance (a few large ratios dominate and destabilize). AReaL’s decoupled PPO plus a staleness bound $\eta$ operationalizes exactly this.

(c) Why even synchronous runs need it. Because the tokens are sampled by the inference engine (vLLM/SGLang) but the log-probs in the loss are recomputed by the training engine (FSDP/Megatron). Different kernels/precision/attention mean ${\pi }_{\text{infer}}\ne {\pi }_{\text{train}}$ for identical weights — a hidden off-policy gap. Truncated Importance Sampling reweighting by ${\pi }_{\text{train}}/{\pi }_{\text{infer}}$ reconciles them. So IS correction is fundamental to engine-disaggregated RL, not just to asynchrony.

---

6. The plumbing: KV-cache reuse and verifier/sandbox infra

Two systems components deserve their own callout because they’re where a lot of the real wall-clock savings (and correctness risk) live.

KV-cache and prefix reuse. Generation is memory-bound, so the KV cache is the bottleneck. Two engine features matter for RL specifically:

• Paged KV cache / continuous batching (vLLM PagedAttention): pack many concurrent rollouts with heterogeneous lengths without fragmentation — essential because CoT lengths vary wildly across a batch.¹²
• Prefix caching (SGLang RadixAttention): GRPO samples a group of $G$ completions for the same prompt. They share the entire prompt prefix, so you compute the prompt’s KV once and branch $G$ ways — a direct, large saving that scales with group size.¹³ Multi-turn agent rollouts reuse the growing conversation prefix across turns similarly. [established]

Verifier / sandbox infrastructure for RLVR. The reward in reasoning/agentic RL is computed by running code, not a forward pass through an RM. At scale this is its own distributed service:

• Code execution must be sandboxed and isolated (untrusted model-generated code), horizontally scaled (thousands of concurrent executions per step), and bounded (time/memory limits, so a hanging program doesn’t stall the queue).
• Determinism and reproducibility matter: a flaky verifier injects reward noise that RL will happily exploit or be destabilized by.
• Latency of verification can rival generation for tool-heavy/agentic tasks — the verifier becomes a first-class throughput concern, not an afterthought.

The verifier is an attack surface

Every verifier is a reward function, and (lesson 06) the optimizer is an adversary. Model-generated code that exit(0)s the test harness, prints the expected answer format without solving, hard-codes known test cases, or reads the answer from the environment are all observed reward hacks. “Verifiable” reward is only as unhackable as your sandbox is airtight. Frontier RLVR infra invests heavily in hardened, isolated, non-gameable verification — this is real engineering, not a checkbox. [established]

---

7. The open-source framework landscape (Sept 2026)

What each is for, and its distinguishing design. All [established]; treat feature specifics as of Sept 2026.

Framework	Origin	Distinguishing design	Best for
veRL / HybridFlow	ByteDance (arXiv:2409.19256, EuroSys’25)1	Hybrid single+multi-controller; 3D-HybridEngine zero-redundancy resharding; colocated by default, async supported; vLLM/SGLang backends	The de-facto default for reasoning RL; broad algorithm coverage (PPO, GRPO, DAPO, GSPO)
OpenRLHF	community (arXiv:2405.11143)5	Ray + vLLM + DeepSpeed-ZeRO; disaggregated actors; clean RLHF/PPO/GRPO; async variants	Accessible disaggregated RLHF; strong reference implementation
TRL	HuggingFace3	Tight HF-ecosystem integration; GRPO, online DPO, GSPO trainers; async rollout work (“Keep the Tokens Flowing”); vLLM server mode	Smaller-scale, research iteration, HF-native pipelines
NeMo-RL	NVIDIA2	Successor to NeMo-Aligner (archived Nov 2025); Megatron-Core + DTensor/FSDP2 backends; scales to very large models	Megatron-scale training on NVIDIA stacks
AReaL	Ant Research (arXiv:2505.24298, NeurIPS’25)7	Fully asynchronous; decoupled PPO + staleness bound $\eta$; interruptible/partial rollouts; ~2.77× over sync	State-of-the-art async throughput; large reasoning runs
slime	THUDM/Tsinghua14	SGLang-native generation + Megatron training; powers GLM-4.5/4.6	SGLang-centric large-scale RL
ROLL	Alibaba (arXiv:2506.06122)15	Ray-based, agentic-RL oriented, flexible reward/env routing	Large-scale agentic + reasoning RL
SkyRL	Berkeley/NovaSky	skyrl-gym environments + skyrl-agent (arXiv:2511.16108)16; async, agent/long-horizon focus	Agentic / multi-turn / tool-use RL
verifiers + prime-rl	Prime Intellect / Will Brown17	Environments Hub + verifiers library standardizing RL environments; decentralized training (INTELLECT-2)	Reusable environments; community/decentralized RL

How to choose (a heuristic, not a rule)

Single-node or research iteration → TRL. Standard reasoning RL at cluster scale → verl (colocated) or OpenRLHF (disaggregated). Squeezing max throughput on big async runs → AReaL or slime. Megatron/NVIDIA-native huge models → NeMo-RL. Agentic/tool-use/multi-turn with reusable environments → SkyRL or verifiers. The convergent trend across all of them: vLLM/SGLang for generation, FSDP/Megatron for training, a queue in between, and increasingly async.

---

8. On-policy distillation and RL–distillation hybrids

Full RL is expensive; a family of cheaper “RL-adjacent” methods sits between SFT and RLVR, and they’re heavily used in 2025–26 [established].

On-policy distillation. The student generates rollouts from its own distribution; a stronger teacher scores those on-policy samples token-by-token, and the student minimizes reverse-KL to the teacher on states the student actually visits. This fixes SFT’s core defect — SFT trains on the teacher’s trajectories (off-policy for the student, exposure-bias-prone), while on-policy distillation trains on the student’s trajectories with dense per-token teacher supervision. Thinking Machines Lab’s “On-Policy Distillation” report (Kevin Lu et al., Oct 2025) frames it as combining RL’s on-policyness with distillation’s dense reward, at a fraction of RL’s cost;¹⁸ Qwen3’s “strong-to-weak” distillation (arXiv:2505.09388) is a production instance.¹⁹

RL then distill (the DeepSeek-R1 finding). DeepSeek-R1 (arXiv:2501.12948) reported a result that reshaped small-model practice: for small models, SFT-distilling on a large RL’d model’s reasoning traces beats running RL directly on the small model²⁰ — the small model can’t explore its way to the reasoning behaviors, but it can imitate them once a big model has found them. So the pattern is: RL the big model, then distill down. [established]

Rejection sampling / STaR / ReST (generate–filter–finetune). A cheap RL alternative: sample many completions, keep only the correct/verified ones, SFT on them, repeat. STaR (arXiv:2203.14465²¹) and ReST/ReST${}^{EM}$ (arXiv:2312.06585²²) formalize this; it’s essentially “the positive half of a GRPO group, used as SFT data.” DeepSeek-R1’s pipeline explicitly interleaves rejection-sampling SFT with RL stages. It’s off-policy and coarser than RL (no gradient on the negatives, no advantage weighting) but simple and stable. [established]

The spectrum, one axis

SFT-on-teacher-traces → rejection-sampling/STaR → on-policy distillation → full RLVR is a spectrum of increasing on-policyness and reward density, at increasing cost. Frontier pipelines mix all of them: cold-start SFT, RL to discover behaviors, rejection-sampling SFT to consolidate, distill to smaller sizes, more RL. It is rarely “pure RL.”

---

9. Overoptimization and evaluation at frontier scale

Lesson 06 introduced reward overoptimization (Goodhart against a learned RM). At frontier scale two things change: (1) RLVR reduces but does not eliminate the problem — verifiers get hacked (§6), and (2) monitoring becomes a serious operational discipline.

How labs monitor. The standard instrument panel during a run:

• KL to reference, both as a control (the $\beta$ leash) and as the x-axis for reading reward — recall Gao et al.’s $d=\sqrt{\mathrm{KL}}$ parameterization (lesson 06). Rising reward at exploding KL is the overoptimization signature.
• Reward broken out by source (correctness vs format vs length), to catch length/format hacking early — length bias is a notorious GRPO failure (see Dr.GRPO, arXiv:2503.20783²³; DAPO’s overlong-reward-shaping, §10).
• Policy entropy (the entropy-collapse alarm, §10) — arguably the single most-watched RLVR health metric in 2025–26.
• Held-out gold eval as a function of compute/KL, never proxy reward alone — and increasingly pass@k (not just pass@1), because pass@1 can rise while pass@k falls (§10, §11).

Benchmark overfitting is the frontier's overoptimization

At the lab level, the “reward” being Goodharted is often the benchmark itself. RLVR on math/code with public benchmarks as (implicit) targets can produce models that ace MATH/AIME/LiveCodeBench while generalizing less than the numbers suggest. This is why Anthropic publicly noted optimizing Claude 3.7 for real-world coding over competition-math benchmarks, and why “does it transfer?” (§10) is a live worry.²⁴ Treat leaderboard deltas as proxy reward. [reported/established]

---

10. Exploration and the entropy-collapse problem

The defining algorithmic pathology of RLVR [established]: as you train, policy entropy collapses — the model becomes confident and deterministic early, exploration dies, and improvement plateaus. Because RLVR only reinforces trajectories the model can already occasionally sample, killing exploration caps the ceiling.

The mechanism. Cui et al., “The Entropy Mechanism of RL for Reasoning LLMs” (arXiv:2505.22617), report a strikingly tight empirical law: downstream performance $R$ is well-fit by an exponential of entropy,

$R\approx -a\exp (H)+b,$

so performance is bought by spending entropy — and once entropy is exhausted, you’re stuck. They trace collapse to the covariance between action log-prob and advantage (high-advantage, already-likely tokens get up-weighted, sharpening the distribution) and propose Clip-Cov / KL-Cov to selectively brake the tokens driving collapse.²⁵

Mitigations in the wild:

• Clip-higher (DAPO, arXiv:2503.14476). Decouple PPO’s clip bounds — raise the upper clip so low-probability but promising tokens can still get up-weighted, preserving exploration.²⁶ Plus dynamic sampling (drop prompts where all $G$ samples are all-correct or all-wrong — they give zero GRPO advantage), token-level loss, and overlong reward shaping. DAPO is the reference recipe for keeping entropy alive.
• KL-penalty removal / tuning. Many reasoning-RL recipes drop or shrink the KL-to-reference term (it can cause premature collapse by pinning toward a low-entropy SFT init) — a notable departure from classic RLHF.
• pass@k-aware objectives and diversity rewards — optimize for the whole group’s coverage, not just the top sample.

Quiz 3: What is entropy collapse in RLVR, why is it fatal to further learning, and how does DAPO's "clip-higher" counter it?

Answer

What it is. During RLVR, policy entropy $H$ falls rapidly — the model concentrates probability on a few trajectories and becomes near-deterministic early in training. Empirically performance tracks entropy tightly ($R\approx -a{e}^H+b$, Cui et al. 2025): you spend entropy to buy accuracy, and once it’s gone, gains stop.

Why fatal. RLVR is reinforcement of self-generated successes — it can only up-weight correct trajectories the policy still samples with nonzero probability. If exploration has collapsed, the policy stops discovering new correct behaviors; it just sharpens around what it already does. So pass@1 may creep up while pass@k stalls or drops (diversity is gone), and the reasoning ceiling is capped by the pre-collapse distribution. Collapse is a self-reinforcing loop: sharpening raises the log-prob of already-likely tokens, whose positive advantage sharpens them further (the log-prob×advantage covariance in the entropy mechanism).

How clip-higher helps. In PPO/GRPO the clip bounds the IS ratio to $[1-ϵ,1+ϵ]$. A symmetric clip caps how much a low-probability token can be up-weighted — so promising-but-rare exploratory tokens get their gradient throttled exactly when you want to encourage them, accelerating collapse. DAPO decouples the bounds, raising the upper clip (${ϵ}_{\text{high}}>{ϵ}_{\text{low}}$) so low-prob tokens with positive advantage can be boosted more, keeping mass on exploratory continuations and slowing entropy decay. Paired with dynamic sampling (discard zero-advantage all-same groups) and token-level loss, it measurably sustains entropy and raises the final ceiling.

---

11. Multi-turn and agentic RL

The 2025–26 frontier is RL for agents: tool use, code execution, search, computer/terminal use, multi-turn tasks. This breaks several comfortable assumptions of single-turn RLVR [established, fast-moving].

What’s different:

• The trajectory is now interactive. A rollout interleaves model tokens with environment observations (tool outputs, search results, execution traces). The environment is part of the MDP, and its latency/flakiness is part of your throughput and reward noise.
• Credit assignment over tool calls. Reward is typically sparse and terminal (task solved or not) but the trajectory spans many turns and tool calls. Which call deserves credit? Options: pure trajectory-level (GRPO over whole trajectories — simple, high-variance), turn-level/step rewards, or process reward models for agents (PRM-style, but now over actions — hard to build and hackable). Mostly unsolved; outcome-level dominates in practice because it’s the only reliably-groundable signal. [open]
• Environments as the new bottleneck. You need many reproducible, isolated, parallel environments. This spawned an ecosystem: SkyRL’s skyrl-gym, Prime Intellect’s Environments Hub + verifiers library, ROLL’s env routing, SWE-bench/terminal-bench harnesses repurposed as RL environments. Building good environments (not algorithms) is increasingly the gating work.

Representative work: ReTool (arXiv:2504.11536, RL for interleaved code-tool reasoning)²⁷, Search-R1 (arXiv:2503.09516, RL for search-augmented reasoning)²⁸, ToolRL (arXiv:2504.13958, reward design for tool use)²⁹, RAGEN/StarPO (arXiv:2504.20073) — which named the “Echo Trap,” a multi-turn instability where the agent collapses onto a repetitive strategy (a multi-turn cousin of entropy collapse).³⁰ Kimi K2 (arXiv:2507.20534) is a prominent agentic post-training effort.³¹

Multi-turn is where reward hacking gets creative

Long horizons + tool access = more surface for exploitation: an agent can spam a cheap tool to pad reward, exploit an environment’s non-determinism, or find a “shortcut” that satisfies the checker without doing the task. Long trajectories also strain the systems stack — partial rollouts, per-turn KV reuse, and bounded environment latency all become mandatory, not optional. [established]

---

12. Open research problems

The honest frontier. For each: the question, and where the debate stands [open unless noted].

1. Credit assignment over long CoT. A correct final answer gives one reward for thousands of tokens. Which tokens mattered? Process reward models (PRMs) try to score intermediate steps but are expensive, need step-level labels, and are themselves hackable; outcome reward (ORM) is groundable but maximally sparse. No clean winner; most frontier RLVR uses outcome reward + GRPO’s group baseline and accepts the variance.

2. Reward beyond verifiable domains. RLVR shines where answers are checkable (math, code). Most valuable tasks aren’t — writing, research, open-ended reasoning, judgment. Emerging directions: rubric-based/generative reward models (“Rubrics as Rewards,” arXiv:2507.17746³²), LLM-as-judge rewards, and hybrids that fuse verifiable + model-based rewards (Gemini 2.5’s reported recipe, arXiv:2507.06261³³). The risk is re-importing overoptimization (§9) the moment the reward is learned again.

3. Generalization of RLVR. Does RL on math/code transfer to other capabilities, or overfit to benchmark formats? Mixed evidence: some cross-domain transfer reported (arXiv:2506.19733³⁴, arXiv:2507.00432³⁵), but also strong format/benchmark overfitting (§9). [contested]

4. Does RLVR create new capability or elicit existing capability? The headline debate. Yue et al. (arXiv:2504.13837) argue RLVR mostly sharpens the base model: pass@1 rises but pass@k (large $k$) can cross over and fall below the base model — RL narrows to high-probability correct paths without adding genuinely new ones.³⁶ ProRL (NVIDIA, arXiv:2505.24864) counters that prolonged RL with enough compute and exploration does expand the reasoning boundary (new solutions the base model never produces).³⁷ A 2025 reconciliation (arXiv:2510.04028) frames it as two stages:³⁸ early RL sharpens/elicits, sustained RL with preserved entropy can expand. Not settled — and it’s the question that decides how much to invest in RL vs base models. [contested — the central debate]

5. Sample efficiency & scaling laws for RL post-training. How does performance scale with RL compute, and how should you split compute between pretraining, SFT, and RL? Early results (e.g. “The Art of Scaling RL Compute,” arXiv:2510.13786) fit sigmoidal compute–performance curves for RL (a ceiling, unlike pretraining’s power law) and offer recipes (ScaleRL) for predictable scaling.³⁹ But RL scaling laws are far less mature than pretraining’s Chinchilla-era understanding. [emerging]

6. On-/off-policy and async at scale. How stale is too stale? Token- vs sequence-level IS? Is the critic worth it (PPO/VAPO, arXiv:2504.05118) or is value-free GRPO enough at scale? Live engineering-and-theory questions (§5). [open]⁴⁰

7. RL vs SFT tradeoffs. “SFT memorizes, RL generalizes” (Chu et al., arXiv:2501.17161) is a useful slogan with real evidence,⁴¹ but SFT is cheaper, more stable, and (via distillation) often the right tool for smaller models (§8). The tradeoff is task-, scale-, and budget-dependent, not universal.

Quiz 4 (open problem): State the "does RL create or elicit capability?" debate precisely. What pass@k evidence fuels each side, and why does the answer change how a lab allocates compute?

Answer

The question. When RLVR improves a model, is it (a) eliciting — reweighting toward correct solution paths the base model could already sample, adding no fundamentally new reasoning — or (b) creating — discovering solution behaviors the base model could not produce at any sampling budget?

Evidence for “elicit” (Yue et al., 2504.13837). Measure pass@k across $k$. RLVR raises pass@1 but, at large $k$, the base model’s pass@k can catch up and overtake the RL’d model’s. Interpretation: RL concentrated probability onto a subset of already-reachable correct paths (higher pass@1) while pruning diversity (lower pass@large-k). If every correct RL solution was already in the base model’s support, RL only re-sorted probability mass — elicitation, not creation. This dovetails with entropy collapse (§10): sharpening trades coverage for confidence.

Evidence for “create” (ProRL, 2505.24864). With prolonged RL, enough compute, and exploration preserved (entropy control, KL handling), RL’d models solve problems the base model fails at even at high k — genuinely new capability, expanding the reasoning boundary. The 2025 two-stage view (2510.04028) reconciles: short RL elicits, sustained-and-well-regularized RL can expand.

Why it changes compute allocation. If RL only elicits, the leverage is in the base model (pretraining + data) and RL is a cheap final polish — spend accordingly, and expect a low ceiling from RL alone. If RL creates, then RL compute is a first-class scaling axis (the o-series / “scale RL compute” thesis), justifying massive investment in RL infra (this whole lesson), exploration methods, and long training. The debate is, concretely, “is the next capability jump bought with pretraining FLOPs or RL FLOPs?” — which is why it’s the field’s most consequential open question, and why the honest answer today is “both, stage-dependent, and not fully quantified.”

---

13. State of the field, September 2026 — a grounded synthesis

What frontier labs appear to actually do for post-training, with explicit confidence. Public disclosure is uneven; distinguish carefully.

The convergent recipe (across labs) [reported, high confidence]:

1. Multi-stage, not pure RL. Cold-start SFT → RL (RLVR where checkable) → rejection-sampling SFT to consolidate → more RL/preference-tuning → distillation to smaller sizes. DeepSeek-R1 (arXiv:2501.12948) documents this explicitly; it’s the template others echo.²⁰
2. RLVR is the reasoning engine. For math/code/reasoning, verifiable rewards + a GRPO-family algorithm (value-free, group baseline) is the workhorse. Value-based PPO/VAPO persists where a good critic pays off.
3. RLHF/RLAIF + Constitutional-AI-style preference tuning handle the non-verifiable half (helpfulness, safety, style) — see 08-scaling-rlhf-and-alternatives and Anthropic’s CAI (arXiv:2212.08073⁴²). Increasingly fused with RLVR in one pipeline (Gemini 2.5, arXiv:2507.06261³³).
4. Infrastructure is disaggregated and trending async, vLLM/SGLang generation + FSDP/Megatron training + hardened verifier services (§3–§7).

Lab specifics (confidence varies):

• DeepSeek [established]: GRPO (from DeepSeekMath, arXiv:2402.03300⁴³), the R1/R1-Zero multi-stage pipeline; the most transparent frontier recipe.
• Qwen (Alibaba) [established]: GSPO (sequence-level IS, arXiv:2507.18071¹¹) for Qwen3; strong-to-weak distillation (arXiv:2505.09388¹⁹); “thinking” modes.
• Kimi / Moonshot [established]: k1.5 (arXiv:2501.12599⁴⁴) — value-free RL via online mirror descent, partial rollouts for long-context, length penalties, long→short distillation, all on custom infra; K2 (arXiv:2507.20534³¹) — agentic post-training, rubric-based critic.
• OpenAI o-series / GPT-5 era [reported/stated-at-high-level]: publicly framed as scaling RL on chain-of-thought + test-time compute; specifics undisclosed. The “scale RL compute” thesis (§12.5) is most associated with this line.
• Anthropic Claude [reported]: RLHF/RLAIF + Constitutional AI heritage, extended-thinking models; publicly emphasized optimizing for real-world coding/agentic use over benchmark math (§9).
• Meta Llama [established/reported]: Llama-3 deliberately avoided online RL (DPO-centric) for stability;⁴⁵ Llama-4 reportedly reversed to online RL with a custom async framework — itself a data point that async RL is now table stakes.⁴⁶
• Google Gemini [established]: Gemini 2.5 report (arXiv:2507.06261) describes fusing verifiable and model-based/generative rewards.³³

The live debates, in one place [open/contested]: (1) RL creates vs elicits capability (§12.4) — the big one; (2) on-policy/sync vs off-policy/async at scale — throughput vs soundness (§5); (3) verifiable vs learned rewards for general capability (§12.2); (4) value-free (GRPO) vs value-based (PPO/VAPO) — is the critic worth it (§12.6); (5) how much of 2025–26’s gains are RL vs better base models and data — likely both, stage-dependent, and not cleanly attributed.

What we genuinely don't know

No public scaling law reliably predicts frontier RL post-training returns; no consensus on the create-vs-elicit question; reward design outside verifiable domains is unsolved; and the most detailed public recipes (DeepSeek, Kimi, Qwen) are open labs — the closed frontier (OpenAI, Anthropic, Google) is inferred, not documented. Hold all lab-specific claims loosely and re-verify. The mechanisms in §2–§11 are far more durable than any §13 attribution.

---

14. Practice problems

Problem 1 (systems design): Size and stabilize a disaggregated async RL run

Setup. You’re training a 32B reasoning model with GRPO. Per step: 1024 prompts × group size $G=16$ = 16,384 rollouts, mean CoT length ~8k tokens. On your cluster, one generator replica sustains ~2,000 rollouts/min; one learner does a gradient step over the 16,384 rollouts in ~40s. Verification (code sandbox) averages 0.5s/rollout, parallelizable. Design the system: (a) how many generators to balance one learner, (b) colocated or disaggregated + sync or async, (c) how you bound and correct staleness, (d) two metrics you watch to catch trouble.

Worked discussion.
(a) Balance. One step needs 16,384 rollouts. The learner consumes a step’s worth every ~40s (plus weight-sync overhead). Generators produce 2,000/min ≈ 33/s each, so to feed 16,384 rollouts per ~40s you need ~16,384/(33×40) ≈ 12–13 generator replicas to keep the learner from starving — round up to ~16 to absorb the long-tail straggler (the step waits on the longest 8k+ CoT). Verification at 0.5s×16,384 = ~137 GPU-free CPU-seconds/step, trivially parallelized across a few hundred sandbox workers — not the bottleneck, but isolate it so a hanging program can’t stall the queue.
(b) Topology. At this scale, disaggregated + asynchronous. Colocated-sync would idle the learner during the ~minutes of generation and idle generators during the 40s update — exactly the §2 imbalance. Disaggregate (separate generator and learner pools, queue between) and run async so both stay saturated; use partial/interruptible rollouts (AReaL-style) so the longest CoT doesn’t gate the step.
(c) Staleness. Set a bound $\eta$ (say ≤4 updates) — rollouts older than that are dropped or down-weighted. Apply a decoupled-PPO / truncated-IS correction: reweight by $\min ({\pi }_{\theta }/\mu ,\bar{\rho })$ where $\mu$ is the generating policy. Also apply TIS for the inference-vs-training engine mismatch (§5.2) even setting async aside. Sync weights to generators every $k$ steps via RDMA/NCCL broadcast; smaller $k$ → fresher data, more sync overhead.
(d) Metrics. Policy entropy (entropy-collapse alarm, §10) and held-out gold accuracy / pass@k vs KL (overoptimization + diversity, §9–§10). Bonus third: reward broken out by correctness vs length/format to catch hacking, and generator/learner utilization to catch pool imbalance. The meta-point: throughput (a,b), soundness (c), and health (d) are three separate concerns you must design for simultaneously — that’s what makes frontier RL a systems problem.

Problem 2 (open-ended reward design): Extend RLVR beyond verifiable domains

Setup. RLVR gave you a great math/code model. Leadership wants the same gains on open-ended research writing (literature synthesis, argumentation). There is no unit test for “good synthesis.” Design a reward. What breaks, and how do you monitor for it?

Worked discussion.
The core problem. You’ve lost the thing that made RLVR safe from overoptimization: a groundable, unhackable verifier (§6, §9). Any reward you build here is a learned proxy, so lesson 06’s overoptimization returns in full force — the optimizer will Goodhart whatever you write down.
Candidate rewards, with failure modes. (1) Learned reward model on human preferences — classic RLHF; fails on the human-label bottleneck and verbosity/sycophancy bias (lesson 06 §8.4). (2) LLM-as-judge — cheap and scalable, but judges are hackable (favor length, confident tone, their own style) and can be adversarially exploited by the policy. (3) Rubric-based / generative reward (“Rubrics as Rewards,” arXiv:2507.17746) — decompose “good synthesis” into checkable sub-criteria (cites real sources? represents opposing views? no unsupported claims?), score each, aggregate. This partially re-verifiabilizes the reward: some rubric items (citation exists, quote is accurate) are checkable, shrinking the hackable surface. (4) Hybrid — verifiable rubric items (grounded) + a generative judge for holistic quality, à la Gemini 2.5 (arXiv:2507.06261).
Best design. Push as much of the reward onto verifiable rubric components as possible (citation existence/accuracy, factual claims cross-checked against sources, structural requirements), and use a generative/rubric RM only for the irreducibly subjective residual — then treat that residual exactly like lesson 06: modest KL budget, RM ensembles/variance penalty, and iterated online re-labeling on the current policy’s outputs.
Monitoring. Watch gold quality (fresh human eval) as a function of KL, expect a peak, and stop near it. Watch for the classic hacks — length inflation, citation fabrication (a verifiable sub-check catches this — a reason to include it), sycophantic hedging, judge-style-matching. The honest conclusion: this is an open problem (§12.2). You can narrow the hackable surface with verifiable rubrics but you cannot eliminate it the way math/code RLVR does — which is precisely why “reward beyond verifiable domains” is a frontier research question, not an engineering task.

---

15. Reading order

Systems / infra first, then the science.

1. Framework foundation — HybridFlow/verl: Sheng et al., “HybridFlow: A Flexible and Efficient RLHF Framework” (arXiv:2409.19256) — the single/multi-controller model and 3D-HybridEngine resharding; read for how generation and training are wired.
2. Disaggregated RLHF — OpenRLHF: Hu et al. (arXiv:2405.11143) — Ray + vLLM + DeepSpeed; the reference disaggregated design.
3. Async & off-policy — the two key papers: Noukhovitch et al., “Asynchronous RLHF” (arXiv:2410.18252) for why bounded off-policyness is nearly free, then Fu et al., “AReaL: A Large-Scale Asynchronous RL System” (arXiv:2505.24298) for staleness bounds, decoupled PPO, and interruptible rollouts.
4. The actor–learner ancestor: Espeholt et al., “IMPALA” (arXiv:1802.01561) — V-trace off-policy correction; the pattern everything above descends from.
5. The engine-mismatch subtlety: search “rollout–training mismatch RL LLM truncated importance sampling” and read the “your RL framework is secretly off-policy” analyses (e.g. the FP16-precision remedy, arXiv:2510.26788).
6. Entropy & exploration: Cui et al., “The Entropy Mechanism of RL for Reasoning LLMs” (arXiv:2505.22617) and DAPO (arXiv:2503.14476) — the collapse law and clip-higher.
7. The central debate: Yue et al., “Does RL Really Incentivize Reasoning Beyond the Base Model?” (arXiv:2504.13837) vs ProRL (arXiv:2505.24864); then the two-stage reconciliation arXiv:2510.04028.
8. Scaling RL compute: “The Art of Scaling RL Compute” (arXiv:2510.13786) — sigmoidal curves, ScaleRL.
9. On-policy distillation: Thinking Machines Lab, “On-Policy Distillation” (Oct 2025 report) — the cheap RL-adjacent middle ground.
10. Lab recipes as case studies: DeepSeek-R1 (arXiv:2501.12948), Kimi k1.5 (arXiv:2501.12599), Gemini 2.5 (arXiv:2507.06261) — read for the pipelines, not the benchmark tables.
11. Agentic RL: ReTool (arXiv:2504.11536), Search-R1 (arXiv:2503.09516), RAGEN/StarPO (arXiv:2504.20073); browse the Prime Intellect Environments Hub and SkyRL for the environment ecosystem.

---

16. What’s next

You’ve reached the end of the curriculum — rl-for-llms. From foundations (MDPs → policy gradients → PPO) through the LLM bridge (token MDP → RLHF) to SOTA (DPO → RLAIF → RLVR → this capstone), you now hold the whole arc. The frontier from here forks into deeper threads, each worth its own lesson — request whichever pulls hardest:

1. agentic-rl-deep-dive (not yet written) — multi-turn credit assignment, environment design, and the emerging “environments-as-the-product” ecosystem (SkyRL, Environments Hub). §11 is only the on-ramp.
2. rl-infra-engineering (not yet written) — a hands-on build of a disaggregated async trainer: weight resharding, TIS in code, KV-cache/prefix reuse, verifier sandbox hardening. Turn §3–§6 into an implementation.
3. scaling-laws-for-rl (not yet written) — the emerging science of RL compute allocation (§12.5): sigmoidal curves, RL-vs-pretraining FLOP tradeoffs, and the create-vs-elicit question as a scaling problem.
4. reward-modeling-beyond-verifiable (not yet written) — generative/rubric reward models, LLM-as-judge robustness, and how labs are extending RL past math/code (§12.2, Problem 2).

And to consolidate: re-read 09-rl-for-reasoning with §5 and §10 in mind — GRPO’s group baseline, the KL term, and the clip bounds all look different once you see them as systems and exploration choices, not just estimator choices.

---

Topic hub: rl-for-llms | Builds on: 09-rl-for-reasoning, 06-rlhf-pipeline, 04-ppo | Concepts: ppo, kl-regularization-rlhf, generalized-advantage-estimation | Filed: 2026-09-02

---

References

Footnotes

1. Sheng et al., “HybridFlow: A Flexible and Efficient RLHF Framework” (veRL), 2024 (EuroSys 2025). arXiv:2409.19256. (Hybrid single/multi-controller programming model; 3D-HybridEngine zero-redundancy resharding; ~70% of DAPO/Qwen-32B step time in rollout generation — profiling figure, approximate and workload-dependent.) ↩ ↩² ↩³ ↩⁴

2. NVIDIA NeMo-RL documentation and repository (successor to NeMo-Aligner), 2025 (https://github.com/NVIDIA-NeMo/RL). [The specific “~65–72% of step time in generation” figure is not pinned to a single citable primary doc; treat as an approximate, workload-dependent profiling number in the same range as veRL/TRL reports.] ↩ ↩²

3. HuggingFace TRL, “No GPU left behind: Keep the Tokens Flowing” (async rollout) blog post and TRL documentation, 2025 (https://huggingface.co/blog). (Framing of async rollouts around the learner-starvation observation.) ↩ ↩²

4. Espeholt et al., “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures,” 2018. arXiv:1802.01561. (Actor–learner decomposition; V-trace off-policy correction.) ↩

5. Hu et al., “OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework,” 2024. arXiv:2405.11143. (Ray + vLLM + DeepSpeed-ZeRO disaggregated actors.) ↩ ↩²

6. Noukhovitch et al., “Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models,” 2024 (ICLR 2025). arXiv:2410.18252. (Bounded generation–training lag is nearly free in quality with large speedups.) ↩

7. Fu et al., “AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning,” 2025 (NeurIPS 2025). arXiv:2505.24298. (Fully asynchronous; staleness bound $\eta$; decoupled/staleness-enhanced PPO; interruptible/partial rollouts. Abstract reports “up to ~2×” speedup; the “~2.77×” figure cited here corresponds to a specific experimental configuration — not confirmed verbatim from the abstract; verify against the paper’s tables.) ↩ ↩²

8. Zhang et al. (Kwai-Klear), “RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning,” 2025. arXiv:2507.07451. (Replays verified successful trajectories for faster convergence.) ↩

9. Li et al., “RePO: Replay-Enhanced Policy Optimization,” 2025. arXiv:2506.09340. (Retrieves off-policy replay samples to improve GRPO efficiency.) ↩

10. Qi et al. (Sea AI Lab / NUS), “Defeating the Training-Inference Mismatch via FP16,” 2025. arXiv:2510.26788. (Roots the inference-vs-training engine mismatch in BF16 rounding error; FP16 largely eliminates it; context for Truncated Importance Sampling.) ↩

11. Zheng et al. (Qwen Team), “Group Sequence Policy Optimization” (GSPO), 2025. arXiv:2507.18071. (Sequence-level importance ratio/clipping; lower-variance but coarser than token-level; stabilizes MoE RL; used for Qwen3.) ↩ ↩²

12. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM), 2023 (SOSP 2023). arXiv:2309.06180. (Paged KV cache / continuous batching.) ↩

13. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (RadixAttention), 2023. arXiv:2312.07104. (Prefix caching via RadixAttention — shared prompt-prefix KV reuse across a group of completions.) ↩

14. THUDM/Tsinghua, “slime” RL framework repository, 2025 (https://github.com/THUDM/slime). (SGLang-native generation + Megatron training; used for GLM-4.5/4.6.) ↩

15. Wang et al. (Alibaba), “Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library” (ROLL), 2025. arXiv:2506.06122. (Ray-based, agentic-RL oriented, flexible reward/env routing.) ↩

16. Cao et al. (NovaSky / UC Berkeley / Anyscale), “SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent,” 2025. arXiv:2511.16108. (Async dispatcher; skyrl-gym/skyrl-agent; SA-SWE-32B lifts Qwen3-32B 24.4%→39.4% on SWE-Bench Verified.) ↩

17. Prime Intellect, “INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning,” 2025 (arXiv:2505.07291), plus the verifiers library and Environments Hub (https://github.com/willccbb/verifiers). (Standardized RL environments; decentralized training.) ↩

18. Thinking Machines Lab (Kevin Lu et al.), “On-Policy Distillation,” Oct 2025 (https://thinkingmachines.ai/blog/on-policy-distillation/). (Student trains on its own rollouts scored token-by-token by a teacher; combines RL’s on-policyness with distillation’s dense reward.) ↩

19. Yang et al. (Qwen Team), “Qwen3 Technical Report,” 2025. arXiv:2505.09388. (Strong-to-weak distillation; thinking modes.) ↩ ↩²

20. DeepSeek-AI (Guo et al.), “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025. arXiv:2501.12948. (Multi-stage cold-start SFT → RL → rejection-sampling SFT → RL pipeline; RL-then-distill beats small-model RL.) ↩ ↩²

21. Zelikman et al., “STaR: Bootstrapping Reasoning With Reasoning,” 2022. arXiv:2203.14465. (Generate–filter–finetune on self-generated correct rationales.) ↩

22. Singh et al., “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models” (ReST${}^{EM}$), 2023. arXiv:2312.06585. (Expectation-maximization-style rejection-sampling self-training; cf. the original ReST, Gulcehre et al., arXiv:2308.08998.) ↩

23. Liu et al., “Understanding R1-Zero-Like Training: A Critical Perspective” (Dr. GRPO), 2025. arXiv:2503.20783. (Length and difficulty normalization biases in GRPO.) ↩

24. Anthropic, “Claude 3.7 Sonnet” announcement and model card, Feb 2025 (https://www.anthropic.com/news/claude-3-7-sonnet). (Publicly emphasized optimizing for real-world/agentic coding over competition-math benchmarks.) ↩

25. Cui et al., “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models,” 2025. arXiv:2505.22617. (Empirical law $R\approx -a{e}^H+b$; log-prob×advantage covariance drives collapse; Clip-Cov / KL-Cov.) ↩

26. Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” 2025. arXiv:2503.14476. (Clip-Higher, dynamic sampling, token-level loss, overlong reward shaping.) ↩

27. Feng et al., “ReTool: Reinforcement Learning for Strategic Tool Use in LLMs,” 2025. arXiv:2504.11536. (RL for interleaved code-tool reasoning.) ↩

28. Jin et al., “Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning,” 2025. arXiv:2503.09516. ↩

29. Qian et al., “ToolRL: Reward is All Tool Learning Needs,” 2025. arXiv:2504.13958. (Reward design for tool use.) ↩

30. Wang et al., “RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning” (StarPO), 2025. arXiv:2504.20073. (Names the multi-turn “Echo Trap” instability.) ↩

31. Kimi Team (Moonshot AI), “Kimi K2: Open Agentic Intelligence,” 2025. arXiv:2507.20534. (Agentic post-training; rubric-based critic.) ↩ ↩²

32. Gunjal et al., “Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains,” 2025. arXiv:2507.17746. (Rubric-based reward via GRPO; up to ~31% relative gain on HealthBench over Likert LLM-judge baselines.) ↩

33. Comanici et al. (Google DeepMind), “Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” 2025. arXiv:2507.06261. (Describes fusing verifiable and model-based/generative rewards.) ↩ ↩² ↩³

34. Hu et al., “Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?,” 2025 (ICLR 2026). arXiv:2506.19733. (RLVR gains generalize inconsistently and can vanish on domains with different reasoning patterns.) ↩

35. Huan et al., “Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning,” 2025. arXiv:2507.00432. (RL-tuned models transfer across domains better than SFT-tuned ones.) ↩

36. Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?,” 2025 (NeurIPS 2025). arXiv:2504.13837. (RLVR raises pass@1 but not pass@k at large $k$; the elicit-not-create argument.) ↩

37. Liu et al. (NVIDIA), “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models,” 2025. arXiv:2505.24864. (Prolonged, well-regularized RL expands the reasoning boundary.) ↩

38. Yao et al., “The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View,” 2025. arXiv:2510.04028. (Reconciles elicit vs. create as early-exploitation then late-exploration phases.) ↩

39. Khatri et al., “The Art of Scaling Reinforcement Learning Compute for LLMs” (ScaleRL), 2025. arXiv:2510.13786. (Sigmoidal compute–performance curves for RL; ScaleRL recipe; >400k GPU-hours of experiments.) ↩

40. Yuan et al. (ByteDance Seed), “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks,” 2025. arXiv:2504.05118. (Value-based framework competitive with / beating value-free GRPO/DAPO on long-CoT.) ↩

41. Chu et al., “SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training,” 2025. arXiv:2501.17161. ↩

42. Bai et al. (Anthropic), “Constitutional AI: Harmlessness from AI Feedback,” 2022. arXiv:2212.08073. (RLAIF / CAI for the non-verifiable half of post-training.) ↩

43. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” 2024. arXiv:2402.03300. (Introduces GRPO.) ↩

44. Kimi Team (Moonshot AI), “Kimi k1.5: Scaling Reinforcement Learning with LLMs,” 2025. arXiv:2501.12599. (Value-free online mirror descent; partial rollouts for long-context; length penalties; long2short.) ↩

45. Grattafiori et al. (Meta), “The Llama 3 Herd of Models,” 2024. arXiv:2407.21783. (DPO-centric post-training; deliberately avoided online RL for stability.) ↩

46. Meta AI, “The Llama 4 herd” announcement, Apr 2025 (https://ai.meta.com/blog/llama-4-multimodal-intelligence/). [The specific claim that Llama-4 “reversed to online RL with a custom async framework” is reported/inferred from public communications, not documented in a peer-reviewed source — treat as [reported], not [established].] ↩