本文へスキップ
Kitbay
スキル

hillclimb-skill

MIT2026年11月10日 更新230GitHubスタースクリプトあり

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks.

スキルの内容

A skill from Caliper, a tool for evaluating agent skills, that improves an existing SKILL.md against its evaluation one change per round. It needs an evaluation with at least eight tasks. After asking for the goal (a higher score or fewer tokens), the part to change, the smallest gain worth keeping and a round budget, it runs a baseline twice to measure noise and holds out a third of the tasks. Each round reads failures from the remaining tasks, makes the smallest general fix and keeps it only if the held-out tasks improve too. It finishes by restoring the highest-scoring version on held-out tasks, comparing it with the baseline, and leaving the edits uncommitted.

できること

  • Measure an evaluation's noise floor
  • Split tasks into training and held-out sets
  • Make and test one skill change per round
  • Report held-out results against the baseline