Mochi is the name of Open Athena's Marin 535B-A23B, the hero run that began in August 2026 on NVIDIA GB200 NVL72 racks donated through the Jen-Hsun and Lori Huang Foundation (about 11 racks). Main pre-training finished on November 5 after about 18T tokens, and on November 17 Open Athena released the run's 57 pre-training checkpoints as BF16 weights on HuggingFace, one branch each from step 6,000 to the final step 393,491, ten of them from the long-context extension. The main branch holds the final checkpoint. These are base checkpoints only, with no instruction tuning or safety post-training. Open Athena says post-training and a technical report will follow, and its release note sets out why it judged the benefit to researchers to outweigh the risk of publishing pre-training artifacts at this scale. The license is OpenMDW 1.1.

The model stores 535.5B parameters with about 22.76B active non-embedding parameters per token. From the config and the architecture post:

  • Shape: 48 layers at hidden size 6,144, shallow by design (width-to-depth ratio of 128), with no leading dense layers and a 128,256-token vocabulary.
  • MoE: a normalized latent MoE (2× compression, with a learnable RMSNorm on the latent) with 8 of 384 routed experts plus 2 shared experts. A third of MLP neurons sit in shared experts. The router uses sigmoid scores, top-k quantile balancing and expert parallelism over a custom all-to-all kernel.
  • Attention: 3:1 sliding-window (2,048) to global layers, partial RoPE on local and none on global layers, gated attention, exclusive self-attention (XSA), QK RMSNorm with a fixed multiplier, and short convolutions on keys, attention output and MLP output.
  • Training: the Hyperball optimizer (MuonH on matrices, AdamH and Adam elsewhere), 46.1M tokens per step, and a data-mix change near 5T tokens. Context grew from 4K to 16K, 65K and finally 262K, the last stage using log scaling of global-attention queries.

The run is live-streamed: loss is published against a forecast preregistered from a scaling ladder, and the team reports no loss spikes. The training data comes from the Datakit pipeline, about 23T deduplicated tokens drawn from 152 HuggingFace datasets. Open Athena publishes no benchmark scores yet. Shifts in evaluation loss, along with concerns raised by a Stanford student, led the team to find substantial leakage of common pre-training benchmarks that its 13-gram decontamination had missed. That limits which evaluations it can fairly report. Serving needs Marin's vLLM fork, and the card says inference parity has not been validated.

Model Details

Architecture MOE
Parameters 535B
Active params 23B
Experts 384 (top-8)
Context window 262,144
Training tokens 18T
License OpenMDW 1.1
open-weightopen-sourcemoepretrainingresearch

Related