NCP-ArchPreview

A latent-space language model from the Intern-NCP Team (Shanghai AI Lab with SJTU's LUMIA lab) that trains on Next Concept Prediction alongside ordinary next-token prediction. The model builds a product-quantized concept vocabulary directly from its own hidden states, predicts future multi-token concepts with a dedicated Concept Module, and feeds those predictions back to guide token generation, with both objectives trained end to end. At 8.9B parameters on 5.73T Dolma-3 tokens the report calls it the largest latent-space LM demonstrated so far, and reports that it reaches OLMo-3-7B's final pretraining loss with 51.3% of the tokens, then outperforms it after full pretraining (a 2.45-point macro-average gain, 5.99 on GSM8K). Released with stage-1 checkpoints every 100K steps, drafters, and the final stage-1 and stage-2 weights.