AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Microsoft Research (with intern co-first authors from POSTECH, KAIST and SUSTech) frames harness optimization as offline mini-batch learning. Each AutoSaddler iteration evaluates the current harness on a mini-batch of tasks, runs a Diagnosis–Patch session that debugs the failing execution traces against the agent codebase and emits a structured code patch (capability patches such as new tools, implementation fixes and agent-loop changes; steering patches such as prompt rules, tool descriptions and PreToolUse hooks), verifies the patch on the same mini-batch, checks generalization on a dev split, and records lessons in EvoDAG, a DAG of harness lineage that seeds the next candidate. The three optimizer agents are built on the Claude Agent SDK with Claude Opus 4.6.

On held-out task groups, AutoSaddler lifts the GAIA2 default ReAct agent from 53.0% to 62.0% Pass@1 (+9.0 pp), SWE-agent on SWE-Bench Pro from 37.3% to 46.9% (+9.6 pp) and Terminus 2 on Terminal-Bench 2.0 from 40.0% to 50.0% (+10.0 pp) — also beating the expert-tuned Terminus KIRA (47.5%) and the automated baselines GEPA and Meta-Harness by 4.4–7.4 pp. It reaches 72.3% GAIA2 dev accuracy after ~1,000 rollouts and its best score from 147 leveraged traces, ~10× fewer than Meta-Harness; harnesses optimized with Opus 4.6 still gave +5.6 pp when the task agent was swapped to Haiku 4.5. Ablations isolate three ingredients: deep debugging over shallow reflection, targeted edits over unconstrained rewriting, and generalization-aware selection over trajectory-specific repair. Code under MIT.