Skip to content
Kitbay
Skill

hillclimb-skill

MITUpdated on Nov 10, 2026230GitHub starsIncludes scripts

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks.

What the skill teaches

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks. After asking for the goal (a higher score or fewer tokens), the part to change, the smallest gain worth keeping and a round budget, it runs a baseline twice to measure noise and holds out a third of the tasks. Each round reads failures from the remaining tasks, makes the smallest general fix and keeps it only if the held-out tasks improve too. It finishes by restoring the highest-scoring version on held-out tasks, comparing it with the baseline, and leaving the edits uncommitted.

Capabilities

  • Measure an evaluation's noise floor
  • Split tasks into training and held-out sets
  • Make and test one skill change per round
  • Report held-out results against the baseline