ACES / NVIDIA SkillEvaluator: Agentic Continuous Evaluation of Skills

NVIDIA's answer to a harness-era review problem: agent "skills" (reusable instruction/tool/workflow packages for Claude Code, Codex, Cursor and other harnesses) are usually gated by scanning their files for structure, style and security, which never answers whether the package actually helps a live agent. ACES (Agentic Continuous Evaluation of Skills) is a repository-native framework that treats a skill as an executable artifact: it runs paired live trials with and without the target skill under the same model, sandbox and grading policy, normalizes the resulting trajectories into an Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics (accuracy, goal accuracy, behavior check, skill execution and skill efficiency among them, plus bring-your-own graders), and reports Skill Lift — the with-skill reward minus the baseline reward for a fixed task, harness, workspace and scorer. The same protocol compares baseline, single-skill, bundle, team-skill and plugin targets against product-owned task suites, and layers on five review dimensions (security, correctness, discoverability, effectiveness, efficiency).

On 145 real skills from internal NVIDIA enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure something different from runtime value (structural score vs. LLM-judge Spearman ρ = 0.14). Across 947 scored paired cases from 58 of 64 production skills on four primary harnesses, mean composite Skill Lift is 0.2134 (95% CI [0.1967, 0.2301]) and outcome-only lift (mean of accuracy and goal accuracy) 0.1799, with composite lift positive in 72.8% of paired cases; the largest gains show up in process metrics — skill execution, behavior check and efficiency — i.e. discovery, routing, workflow following and tool use that document scans cannot observe. Versions were accepted at Agent Skills '26 (ACM CAIS 2026) and the KDD 2026 Enterprise AI Agents workshop (oral). The methodology ships as the Apache-2.0 NVIDIA SkillEvaluator repo (multi-tier evaluation with quality gates and semantic-overlap detection; ~350 stars).