RL Post-Training Infrastructure

Definition

RL post-training at frontier scale is a distributed-systems problem wearing an optimization problem’s clothes. An RL step is a pipeline — generate → score → update — and generation dominates wall-clock (~65–90% for long-CoT reasoning; measured in veRL, NeMo-RL, TRL). The algorithm (GRPO/PPO) is settled enough that throughput — how fast you turn GPUs into fresh, correctly-scored rollouts — is what gates progress. Every architectural choice follows from “make generation not be the bottleneck.”

How it works

A step is $\underset{\text{1. generate (autoregressive decode)}}{\underbrace{y\sim {\pi }_{\theta }(\cdot \mid x)}}\to \underset{\text{2. score (sandbox/checker)}}{\underbrace{r=\text{verify}(x,y)}}\to \underset{\text{3. update (one backward)}}{\underbrace{\theta \leftarrow \theta +\eta \hat{g}}}$. Generation is memory-bandwidth-bound sequential decoding ($O(\text{prompts}\times G\times T)$ steps); the update is one compute-bound parallel pass. They don’t want the same machine — hence the generation-vs-training split.

The converging architecture: disaggregated, asynchronous actor–learner (the IMPALA lineage, Espeholt et al. 2018, arXiv:1802.01561):

• Generators (vLLM/SGLang replicas) hold possibly-stale weights ${\theta }_{\text{behavior}}$, decode with paged KV-cache + prefix reuse (GRPO’s $G$ completions share the prompt prefix — compute it once, branch $G$ ways).
• Verifier layer — sandboxed, isolated, horizontally-scaled code exec / math checkers / tool environments; an attack surface (the optimizer will exit(0) the harness, hard-code tests) and a first-class throughput concern.
• Queue/buffer holds scored rollouts tagged with staleness (age in updates).
• Learner (FSDP/Megatron) pulls bounded-staleness batches, applies an IS correction, updates, and periodically broadcasts fresh weights.

Colocated vs disaggregated is the placement fork (orthogonal to sync/async): colocated time-shares GPUs (veRL HybridFlow, arXiv:2409.19256, with 3D-HybridEngine zero-redundancy resharding); disaggregated gives each its own pool and runs concurrently/async (OpenRLHF, arXiv:2405.11143). The 2025–26 drift for large runs is disaggregated + async.

Async, staleness, and importance sampling

Asynchrony breaks PPO/GRPO’s on-policyness: rollouts come from a stale behavior policy $\mu$. The fix is an importance-sampling correction, truncated to control variance:
${\hat{g}}_{\text{corrected}}={\mathbb{E}}_{y\sim \mu }[\min (\rho ,\bar{\rho })\hat{A}(x,y){\nabla }_{\theta }\log {\pi }_{\theta }(y\mid x)],\rho =\frac{{\pi }_{\theta }(y\mid x)}{\mu (y\mid x)}.$
Lineages: V-trace (IMPALA, truncated IS on the value target) and decoupled PPO (AReaL, arXiv:2505.24298 — explicit staleness bound $\eta$, interruptible rollouts, ~2.77× over sync). A modest lag is nearly free (Noukhovitch et al., arXiv:2410.18252) — the regime is near-on-policy, not replay-buffer off-policy. Subtle but load-bearing: even synchronous runs are secretly off-policy because tokens are sampled by the inference engine but log-probs recomputed by the training engine (${\pi }_{\text{infer}}\ne {\pi }_{\text{train}}$ for identical weights) — fixed by Truncated Importance Sampling (${\pi }_{\text{train}}/{\pi }_{\text{infer}}$, e.g. FP16 remedy arXiv:2510.26788).

Why it matters

Throughput gates frontier RL, so infra is the research: the frameworks (veRL, OpenRLHF, TRL, NeMo-RL, AReaL, slime, SkyRL, verifiers/prime-rl) converge on vLLM/SGLang generation + FSDP/Megatron training + a queue between + increasingly async. The through-line: three roles (generate/train/verify) have different optimal hardware, so disaggregate and keep every pool saturated. Monitoring is operational discipline — policy entropy (collapse alarm), held-out gold/pass@k vs KL (overoptimization), reward broken out by source (hacking). Llama-4 reportedly reversed Llama-3’s DPO-centric stance to online async RL — a signal that async RL is now table stakes.

Taught in

• 10-frontier-and-infra — the whole systems capstone: gen-vs-train split, disaggregated async actor–learner, staleness/IS, verifier infra, frameworks.

See also

• grpo — the $G$-sample algorithm whose rollouts dominate cost
• rlvr — verifier/sandbox infra as reward
• agentic-rl — environments and partial rollouts for multi-turn
• ppo — the IS ratio/clip that async reuses
• reward-overoptimization — what the monitoring panel watches for
• rl-for-llms