HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

ByteDance Seed (with SUTD, Georgia Tech and M-A-P) shifts the unit of agent evaluation from task outputs to runnable infrastructure: since a model's capability increasingly depends on its model-external execution harness, can the model build that harness itself? In Creation, the agent starts from a minimal seed and a few cases and must construct a complete execution system; in Evolution, it iteratively revises its own harness using downstream execution feedback. Constructed harnesses are scored on capability (held-out task success) and efficiency (execution-token cost) across six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 instances, with hidden evaluation tasks withheld from development.

Generated harnesses remain substantially behind mature human-engineered references on code and on search/research, while matching or exceeding them on writing and ML experimentation — with large variation in execution cost. The Evolution-stage caution: visible and held-out gains move in the same direction only 53.1% of the time (34 of 64 version switches), and only 2/9 declared final versions are held-out optimal — visible feedback works for local search but is unreliable for final selection, favoring lucky runs and amplifying overfitting. Part of the Seed team's three-benchmark Self-Developing Agents program (with Aspire and S³Gym) probing the path from "half-loop" recursive self-improvement — which assumes a golden verifier — to closed-loop RSI.