Fifteen Crashed Runs, Two Merged GRPO PRs, and the End of the Post-Training Wild West
Notes (and my rant) on Q1 2025's RL post-training scramble, and what the field consolidated into.
The March 18 graveyard
Here is what one day of GRPO training looked like on March 18, 2025.
I was trying to GRPO-train Llama 3.1 8B Instruct on a math task with unsloth's kernels. My wandb project for that day: 23 runs, 15 crashed, 5 finished, 3 failed. Same hyperparameters (learning rate 5e-6, batch size 32, one epoch), tortured in slightly different configurations until enough of them stayed alive long enough to log a curve.
This was normal. Everyone reproducing R1 that spring had a graveyard just like it.
The thing you actually learn from a graveyard is that at 7B+ scale, the model is not what fails. Serialization fails. vLLM's CUDA graph capture fails when your prompt lengths cross a boundary. The reference model gets loaded twice by accident and you OOM. Reward normalization silently collapses to zero because your batch happened to be all-correct or all-wrong. The unsloth wrapper drops a gradient. Your rollout worker deadlocks on a bad EOS token. You watch nvidia-smi for hours.
None of it is glamorous. All of it is where the actual work is.
Which means the first useful thing you can do in Q1 2025 isn't invent a new algorithm. It's fix the trainer.
PR #3335: exposing eager mode
Merged April 21, 2025.
vLLM's default execution mode is a hybrid CUDA-graph thing. It's faster steady-state but has an ugly warmup cost, and it behaves badly when your prompt-length distribution shifts across a rollout batch. Which is exactly what happens under GRPO. You generate G completions per prompt, they end up slightly different lengths, the CUDA graphs get recaptured constantly, and the VRAM you thought you had for KV cache is now gone.
The workaround is one keyword argument: enforce_eager=True in the vLLM constructor. Pure eager execution. Slower per token, but you actually know how much memory you're using. The difference matters when you're squeezing a rollout onto a single H100.
TRL's vLLM serving script didn't expose the switch. So I added it. One flag, defaulted off for backcompat, wired through. Twelve additions in one file. Landed clean.
It's the smallest change I can imagine making. It's also the change that made a specific class of runs actually work on my hardware. If you can't fit the rollout, you can't train. That is a training-environment problem dressed up as an infra problem, and I would argue most of what people were calling "infra" in early 2025 was actually that.
PR #3434: two-sided clipping
Merged May 13, 2025.
Standard GRPO objective, per token:
ratio = policy_new(token) / policy_old(token)
advantage = (reward - group_mean_reward) / group_std_reward
loss_tok = -min( ratio * advantage,
clip(ratio, 1 - eps, 1 + eps) * advantage )There's an asymmetry that took the community a while to notice. When the advantage is negative (meaning "in retrospect, this token was worse than the group average"), the clip only fires on the lower side of the ratio. Upper side is unbounded. So if the probability ratio grows large for a token you're trying to penalize, loss magnitude explodes and you get an update that yanks the policy hard away from the reference in a single step.
In practice: run looks fine for 400 steps, then loss explodes in one batch.
Prime Intellect's INTELLECT-2 report addresses this with a second cap that fires only when the advantage is negative, so the penalty side can't run away. I ported it into TRL. A new optional bound, a small edit inside the trainer's loss so the second clip fires only when the advantage is negative and the bound is set, and a regression test that constructs an adversarial batch (a token with a large ratio and negative advantage) and asserts the loss stays bounded.
99 additions, 1 deletion, 3 files. Merged the next day. The recommendation in the docstring: set the second bound larger than the first, so normal updates flow through and only pathological ones get clipped.
I read the paper on a Sunday, opened the PR that evening, and it went in the next day. That is the version of this loop that keeps me showing up.
Two-sided clipping is table stakes now. Every serious GRPO derivative (DAPO from ByteDance and Tsinghua, Dr. GRPO from Sea AI Lab, GSPO from the Qwen team, the value-based revival typified by VAPO) either ships some version of it or supersedes it with a different bounded objective.
PR #6237: the argument, not the code
I was going to add support for DOPD (Dual On-Policy Distillation) as its own trainer class in TRL's experimental tree. That's how the sibling PR I was building on had it structured: new trainer, new config, new tests, new entry point.
Then I actually read the code and talked it through with a couple of collaborators, and:
DOPD is not a structurally distinct trainer. It's a config-variant of SDFT.
The reasoning is obvious once you stare at both loops side by side. Both trainers sample on-policy completions from the student given the prompt, score them against a privileged teacher given the prompt plus extra context the student didn't see, take a divergence-based loss over the completion, and backprop through the student. Same data shape, same optimizer step, same rollout structure.
What actually differs is the shape of the divergence: which side's top-k defines the support, whether the divergence runs full-vocab or truncated, and (for DOPD specifically) a per-token router that picks one of four divergences based on how far apart student and teacher probability mass are on that token, and which side is more confident.
TRL splits trainers on loop-shape or data-contract differences. DPO vs KTO vs ORPO. GRPO vs online DPO. Not "the loss function is different but the loop is the same." Adding a whole new trainer for DOPD would have doubled the surface area and made "swap the divergence you use for on-policy distillation" a strictly cross-trainer concern.
So instead of a new trainer, SDFT gets: a fourth distillation mode, one knob for whose top-k defines the support, an optional per-token cap on the divergence, five hyperparameters for the router that only fire in the new mode, and a new routed-loss function that implements the four regimes plus a fallback.
The router is what's interesting. Per token, it looks at (a) how far apart student and teacher probability mass are, and (b) how confident each side is on its top choice, and picks:
Low gap, high confidence. Light top-k reverse-KL. Teacher and I mostly agree; nudge my top-k ranking.
Low gap, low confidence. Stop-gradient self-regularization anchor. Neither of us knows what to do here; don't panic.
High gap, teacher confident. Full-vocabulary JSD. Teacher is sure and disagrees; do a real distillation step across the whole vocab.
High gap, student confident. Light stop-gradient student-consistency nudge. I'm sure and disagree; leave a gentle tether so I don't drift.
Ambiguous (high gap, low confidence on both sides). Falls back to regime 2.
619 additions, 22 deletions, 7 files. Bigger than the first two. The argument matters more than the code.
What in the wild era it was
DeepSeek R1 dropped January 20, 2025. Kimi K1.5 the same day. Within a week, distilled 1.5B–70B variants were outperforming much larger non-reasoning baselines. NVDA lost 17% in a single session on January 27 as the market repriced training compute demand.
That was the starting gun. TRL's GRPO trainer had landed a few weeks earlier. unsloth started shipping GRPO kernels. vLLM's sampling APIs were changing under everyone's feet. verl was months old and rewriting its abstractions weekly.
Nobody had a settled opinion on:
Whether to use vLLM, sglang, or raw HF generate for rollouts.
Whether the reference-model KL should be estimated by a Schulman-style k3 or computed exactly.
Whether reward normalization should be per-batch, per-group, or a running mean.
Whether to clip on the token level or the sequence level.
Whether reward models were still the right primitive, or you should skip them and use rule-based verifiers.
Everyone's stack was different. Everyone's stack broke in different ways. The public wandb projects of that era all look like mine.
The three PRs above are the residue of that period. Someone reads the newest paper (DAPO, DOPD, INTELLECT-2). They notice a stability issue nobody had cared about a month earlier because a month earlier the trainer was even more broken. They open a small PR. Maintainers merge it because they're running the same experiments and hitting the same bugs.
Everyone was a prospector.
Mid-2026: what the field looks like now
Eighteen months later, the shape of post-training work has quietly consolidated. Several things happened at once.
The trainer stack calmed down. verl and OpenRLHF are the two mature choices for production RL post-training at 7B+ scale. TRL is where research prototyping happens. vLLM did a ground-up V1 rewrite in early 2025 that made prefix caching, chunked prefill, and speculative decoding composable by default. SGLang v0.5 is the only open serving stack that reproduces DeepSeek's own reported throughput on large mixture-of-experts models. Disaggregated prefill/decode moved from research (Mooncake, FAST 2025 best paper) to default integration in every serving stack that matters. Nobody argues about eager mode in vLLM help threads anymore. You know when you need it.
Verifiable rewards ate reward models. For structured-output tasks (code, math, tool use, JSON generation, SQL), nobody trains a reward model anymore. You run the code. Check the unit tests. Symbolically verify the math answer. Execute the SQL against a fixture DB. Reward becomes a compiled function, not a learned one. RLHF is now a specific technique for a specific case (preference-shaping on natural-language outputs where verification is subjective), not the default paradigm.
The bottleneck moved. With trainers stabilized and rule-based verifiers cheap to write, the interesting question stopped being "how do I get GRPO to not explode" and became "how do I generate a diverse, task-hard, reward-clean batch of prompts for it to train on." Prompt generation is an RL environments problem. It is the thesis behind Preference Model, the Prime Intellect Environments Hub, Mercor's environments product, Scale's RL team, and the internal environments teams at Anthropic, OpenAI, and Google DeepMind that shipped these ideas years before publishing about them.
Agentic post-training arrived. The 2025 wave was single-turn. Prompt in, completion out, reward on the completion. The 2026 wave is multi-turn agents in real environments: a browser, a code sandbox, a synthetic workplace, an IDE with a repo checked out. Multi-turn changes what "an environment" is. It is not a dataset of prompts with a reward function. It is a simulator with state, tool calls, partial observations, and long-horizon credit assignment. Anthropic went from about a billion dollars in ARR at the start of 2025 to over thirty billion by April 2026, essentially all of it Claude Code and agentic API use. Revenue at that scale is what is directing capex into training-time environment infrastructure right now.
The compute floor dropped hard. A 3B or 7B GRPO run on a self-generated code-execution environment now fits on a two-H100 box for the price of a decent lunch. H100 spot rental collapsed from a peak of about eight dollars an hour in 2024 to roughly a dollar-fifty by late 2025 as the neocloud price war intensified. The training-cost narrative moved with it: DeepSeek V3 at about six million dollars was the January 2025 headline, and MiniMax released their M1 reasoning model six months later for five hundred and thirty-five thousand dollars in total training cost. The most interesting environment work right now is not happening at frontier labs. It's small teams and individuals who can rent a cluster for a weekend, hack together a rule-based verifier, and have verified GRPO running by Monday.
The through-line from March 2025 to now: we stopped writing new losses and started writing better simulators. The scars from the wild-west era (the two-sided clip, the eager-mode fallback, the DOPD router) all live on as small comforts inside a more mature stack. The frontier moved downstream of the trainer, into the environments themselves.
Compute per useful thought is five to ten times cheaper than it was eighteen months ago. Thoughts per useful task are ten to a hundred times more expensive, because reasoning-first models and multi-turn agents both push token consumption up by an order of magnitude or two. Total inference spend went up, not down.
Still climbing.
Housekeeping
Toronto. GitHub: github.com/ucalyptus. Email: hello@ucalyptus.me.
wandb graveyard: wandb.ai/masc/huggingface.
Sayantan (ucalyptus)