Skip to content
Kitbay

agent-observability-build-eval-from-annotations

MITUpdated on Nov 9, 2026185GitHub starsIncludes scripts

A Datadog skill that turns human labels in an LLM Observability annotation queue into an evaluator.

What the skill teaches

A Datadog skill that turns human labels in an LLM Observability annotation queue into an evaluator. The agent reads the queue, finds which part of each trace the labels refer to, drafts an LLM judge, agrees a scoring metric with you and measures the judge against the labelled rows. It then revises the judge one change at a time for a limited number of rounds, checks it on held-out rows and creates the result in Datadog as an evaluator that stays switched off until you enable it. It bundles a local judge runner and a scoring script and can use the Datadog MCP server or the pup CLI.

Capabilities

  • Read labelled interactions from an annotation queue
  • Draft an LLM judge that predicts the human labels
  • Score the judge against existing labels with an agreed metric
  • Refine the judge over several measured rounds
  • Create the final evaluator in Datadog in a disabled state