IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

A UC San Diego-led collaboration with CMU (Aviral Kumar, Ruslan Salakhutdinov, Virginia Smith, Amrith Setlur, Yuxiao Qu) and MBZUAI's Institute of Foundation Models (Taylor Killian, Zhengzhong Liu, and president Eric Xing) that asks the Chinchilla question for RLVR: given a fixed sampling budget C = B_p·n·M, how should it split across parallel rollouts per problem (n), problems per batch (B_p) and sequential update steps (M)? A prerequisite finding is that scaling laws only emerge from a "healthy" recipe — stability is governed by problem difficulty relative to the base model, entropy/KL regularization, and scaling the learning rate with effective batch size — after which reward-vs-compute curves fit smooth sigmoids whose upper envelope defines a compute-optimal frontier.

Main results (GRPO on Qwen2.5-7B-Instruct over Easy/Hard math splits bucketed by pass rate, validated on Qwen3-4B-Instruct and Llama-3.1-8B-Instruct, with PPO and CISPO checks): the compute-optimal n grows predictably with budget, then saturates — on easy problems via solution sharpening, on hard ones via coverage of rare successful trajectories; under a fixed system batch B = B_p·n, the optimum shifts from many-problems/few-rollouts at low compute to few-problems/many-rollouts at high compute; and B_p is mostly a stability knob with weak performance effects in a moderate range. The tabular-bandit prediction (favor sequential steps M) inverts in practice because larger n mitigates gradient interference across problems, which otherwise leaves a fraction of even easy problems at near-zero pass@1. Net effect: RL scaling laws recast as prescriptive allocation rules for sampling compute, complementing pretraining-style compute-optimal analysis on the post-training side.