Skip to content
Kitbay

agent-observability-auto-experiment

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill that runs a measured improvement loop on an LLM application's code using its Datadog LLM Observability data.

What the skill teaches

A Datadog skill that runs a measured improvement loop on an LLM application's code using its Datadog LLM Observability data. The agent loads an evaluation corpus (from an ml_app, a dataset, an annotation queue, a set of trace IDs or a local file) into Datadog Datasets, builds a local scoring harness, records a baseline score, then makes one change at a time and keeps it only when the re-scored result improves. It works through the Datadog MCP server or the pup CLI, asks for its inputs before starting and estimates run cost from the measured baseline. Harness templates in Python and Node are included.

Capabilities

  • Load traces, datasets or annotation queues into Datadog Datasets
  • Build a local evaluation harness and score a baseline
  • Make one focused code or prompt change per iteration and re-score it
  • Keep changes only when the score improves
  • Work through the Datadog MCP server or the pup CLI