"One Success Isn't Reliability." An MCP-compatible stateful tool sandbox and benchmark for tool-agent-user interaction from twelve Microsoft authors (Zhuochun Li et al.): version 1.0 holds 507 tasks across retail, travel, auto insurance, neobank, and IT/HR settings, and scores reliability as repeated success rather than a single pass. A harness-diagnostic evaluation in the τ-bench lineage. Filed late.

Paper

Evaluation Details

Tasks 507
Domains 5
Scoring repeated-success reliability over stateful tool-use episodes
Domains: retail, travel, insurance, banking, IT/HR
evalagentstool-use