Agentic RL (multi-turn, tool-use)
Definition
Agentic RL is RL for interactive, multi-turn tasks — tool use, code execution, search, computer/terminal use — where a rollout interleaves model tokens with environment observations (tool outputs, execution traces, search results). It is the 2025–26 frontier extension of single-turn RLVR, and it breaks several comfortable assumptions: the environment becomes part of the MDP, credit must be assigned across many turns, and building good environments (not algorithms) becomes the gating work.
How it works
The trajectory is now prompt, tool-call, observation, tool-call, …, answer, with the environment inside the loop:
- Credit assignment over tool calls. Reward is usually sparse and terminal (task solved or not), but the trajectory spans many turns. Which call deserves credit? Options: trajectory-level (GRPO over whole trajectories — simple, high-variance, dominant in practice because it’s the only reliably-groundable signal), turn/step rewards, or agent PRMs over actions (hard to build, hackable). Mostly unsolved.
- Environments as the bottleneck. You need many reproducible, isolated, parallel environments — this spawned an ecosystem: SkyRL’s
skyrl-gym, Prime Intellect’s Environments Hub +verifierslibrary, ROLL’s env routing, SWE-bench/terminal-bench harnesses repurposed as RL environments. - Systems strain. Long trajectories require partial/interruptible rollouts, per-turn KV/prefix reuse, and bounded environment latency (the verifier/tool can rival generation cost) — mandatory, not optional. See rl-post-training-infra.
Representative work: ReTool (arXiv:2504.11536, interleaved code-tool reasoning), Search-R1 (arXiv:2503.09516, search-augmented reasoning), ToolRL (arXiv:2504.13958, reward design for tools), RAGEN/StarPO (arXiv:2504.20073) — which named the “Echo Trap,” a multi-turn instability where the agent collapses onto a repetitive strategy (a multi-turn cousin of entropy collapse). Kimi K2 (arXiv:2507.20534) is a prominent agentic post-training effort.
Why it matters
Agentic capability — an LLM that acts in the world across many steps — is where much of the near-term economic value sits, and RL is the natural way to optimize it end-to-end for task success rather than imitation. It is also where reward hacking gets creative: long horizons + tool access give more surface to exploit (spam a cheap tool to pad reward, exploit environment non-determinism, find a checker shortcut that skips the task). The consensus 2026 view: outcome-level credit dominates because it’s the only reliably groundable signal, multi-turn credit assignment is an open problem, and “environments-as-the-product” is an emerging ecosystem where building the env matters more than the algorithm.
Taught in
- 10-frontier-and-infra — §11 multi-turn/agentic RL: interactive trajectories, tool-call credit assignment, environments, the Echo Trap.
See also
- rlvr — the single-turn base agentic RL extends
- grpo — trajectory-level advantage over multi-turn rollouts
- process-reward-model — step/action rewards for agents (hard, hackable)
- rl-post-training-infra — environments, sandboxes, partial rollouts
- chain-of-thought-rl — long-horizon reasoning this builds on
- rl-for-llms