LMMs-Eval

The de-facto standard evaluation harness for large multimodal models (4.3K★): one-for-all benchmarking across text, image, video, and audio tasks — dozens of benchmarks behind a unified lm-evaluation-harness-style interface, plus the LiveBench-style contamination-aware track introduced in the paper. Most open LMM releases (LLaVA line, InternVL, Qwen-VL ecosystem) report numbers produced by it.

Maintained by LMMs-Lab, the open research community anchored by NTU (Ziwei Liu, Bo Li) but formally independent — attribution here is per-item, and this harness is the community's infrastructural core, alongside the LLaVA-OneVision model line and the lmms-engine training stack.