Zum Inhalt springen
Kitbay
Skill

hillclimb-skill

MITAktualisiert am 10. Nov. 2026230GitHub-SterneEnthält Skripte

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks.

Was der Skill vermittelt

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks. After asking for the goal (a higher score or fewer tokens), the part to change, the smallest gain worth keeping and a round budget, it runs a baseline twice to measure noise and holds out a third of the tasks. Each round reads failures from the remaining tasks, makes the smallest general fix and keeps it only if the held-out tasks improve too. It finishes by restoring the highest-scoring version on held-out tasks, comparing it with the baseline, and leaving the edits uncommitted.

Funktionen

  • Measure an evaluation's noise floor
  • Split tasks into training and held-out sets
  • Make and test one skill change per round
  • Report held-out results against the baseline