Modality Maturity Index: A Benchmark for Assessing Multimodal Capabilities of Omni Models

Meta Superintelligence Labs (Rohit Patel, Dieuwke Hupkes, Sloan Strader) proposes the Modality Maturity Index (MMI), a benchmark for the "omni" claim: 893 human-written prompts spanning five modalities (text, image, audio, video, document) and combinations of up to three modalities on both the input and the output side. Each prompt specifies which output modalities a correct answer must contain and carries human-authored rubric criteria per expected modality; the MMI Value averages per-modality rubric scores, while a supplementary Modality Presence Score (MPS) — a per-prompt F1 over the expected output modalities — separates "didn't produce the modality" from "produced it wrong". Existing suites almost all test text plus one modality and never ask whether a model chooses the right output modality.

The headline finding is how immature omni models are on the output side: MPS ranges from 15.6 (Claude Opus 4.6) and 16.3 (Llama 4 Maverick) through 25.4 (Gemini 3.1 Pro) and 27.2 (Gemini 3.1 Flash) to 34.9 (GPT-5.4), with precision near 100 (models rarely emit an unrequested modality) but recall below 35 for every model; input failure rates run from 8% (Gemini 3.1 Pro) to 37% (Claude Opus 4.6); no evaluated model can output audio, and GPT-5.4 is the only one that natively returns images. Because so few assets come back, the rubric-based MMI Value is validated separately on tool-scaffolded models: an LLM judge applying the rubrics agrees with rubric-blind human annotators on 70.8% of judgments (Scott's π 0.41; 66.5% on image to 74.4% on video). Code on GitHub and the dataset on HuggingFace (CC-BY-4.0).