JustRL II

"Scaling Small LLMs to 128K Reasoning with a Critic." The sequel to JustRL (December 2025, arXiv 2512.16649), which argued that a minimal single-stage RL recipe with fixed hyperparameters matches elaborate multi-stage pipelines on a 1.5B model. JustRL II adds a critic and length-adaptive advantage estimation so that small models can be trained to reason over 128K tokens: on the UltraData-RL-2609 set of 85,995 verifiable problems, AIME 2025 rises from 61 to 81 in roughly 300 RL steps, and the recipe produced the final checkpoint of MiniCPM5-2B (86 on AIME 2025). The RL initialization checkpoint, a 2B Llama-architecture model with 65K positions, is on HuggingFace; the write-up is a Notion post with no arXiv version yet.