Rollout Routing Replay (R3)

"Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers" — from LLM-Core Xiaomi (with Peking University; senior author Luo Fuli). Identifies a root cause of RL instability specific to mixture-of-experts models: the rollout engine and the trainer route tokens to different experts, so the recomputed policy probabilities diverge from the true generation distribution and the importance ratio explodes. R3 records the inference engine's routing distributions and replays them during training, cutting training-inference KL divergence from 1.5×10^-3 to 7.5×10^-4 (near dense-model parity), preventing training collapse in every tested configuration, and beating truncated importance sampling by up to +5.6 on math RL and +6.8 Pass@1 on SWE-bench multi-turn tasks (Qwen3-30B-A3B).

Rapidly became a standard tool for open MoE RL: the Marin project's MarinSkyRL stack adopted R3 (paired with truncated importance sampling) as a core ingredient for RL on 30B–80B MoEs, and it anchors the router-alignment line of work that MoE post-training pipelines now build on.