Skip to content
Kitbay
Skill

eval-harness

MITUpdated on Nov 17, 2026305,644GitHub stars

A skill from the ECC collection for eval-driven development in AI coding sessions, treating evals as the unit tests of agent work.

What the skill teaches

A skill from the ECC collection for eval-driven development in AI coding sessions, treating evals as the unit tests of agent work. Before coding, the agent writes capability evals for new behaviour and regression evals against a baseline, stored under .claude/evals; it then implements, runs them and reports pass and fail counts. Graders can be code checks such as tests or builds, a model scoring against a rubric, rule or schema checks, or a human for risky changes such as security. Reliability is tracked as pass@k and the stricter pass^k. Bundled local scripts keep hash-linked records of runs, but executing candidate code is disabled on every system until a reviewed isolation layer exists.

Capabilities

  • Define capability and regression evals before coding
  • Grade results with code, model or human checks
  • Track pass@k and pass^k reliability
  • Write an eval report for a feature