Molt: PyTorch-Native Agentic RL Training Framework

NVIDIA's training framework for agentic reinforcement learning, shipped as a NeMo Labs project with readability as the primary design criterion: ~8.6K Python lines (vs. ~62K for verl) — "compact and clean enough for a researcher to hold in their head." A single asynchronous loop — a Ray queue orchestrating the agent pool, unforked vLLM rollout engines, and an FSDP2/AutoModel policy actor (no hybrid controller, no parameter server, no Megatron dependency) — trains multimodal and mixture-of-experts policies. Its token-first "TITO" contract guarantees every trained token is exactly the token the engine generated: trajectories flow token-id-exact from rollout to training with no re-tokenization, and agents stay ordinary Python — either a Gymnasium-style Env or an unmodified OpenAI/Anthropic-SDK ChatAgent captured through a loopback proxy.

Techniques: a streaming rollout pool with partial rollouts (per-token importance correction when resumed requests mix policy versions), rollout-routing replay so MoE training matches inference-time expert choices, automatic trajectory segmentation under context compaction, and global whole-batch token normalization. Algorithms: REINFORCE++, GRPO, RLOO, PPO with GAE, Dr. GRPO, and on-policy distillation. The same codebase scales from 4B dense to 1T-class MoE by configuration (validated on a 700B-class MoE at EP256); on Qwen3-30B-A3B across 16 GPUs with DAPO-Math prompts it is statistically comparable to a Megatron-Core + SGLang stack (119.4±2.3 vs. 109.5±10.3 s/step), with speculative decoding giving a 5× generation speedup and prefix caching cutting multi-turn re-prefill to 0.05 s. Ships recipes for Qwen3 (dense and MoE), Nemotron-Omni, and GLM-5.2; Apache 2.0. Senior authors include Jan Kautz and Yi Dong. Part of the agent-training execution layer we track alongside OpenForgeRL.