Skip to content
Kitbay

agent-observability-experiment-analyzer

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill for reading the results of offline LLM experiments recorded in Datadog LLM Observability.

What the skill teaches

A Datadog skill for reading the results of offline LLM experiments recorded in Datadog LLM Observability. Given one experiment ID, or a baseline and a candidate, the agent works out what changed, picks metrics, looks for notable signals, investigates individual records and writes a report with findings and recommendations. It can answer a specific question about the runs instead of exploring, and can save the report locally or to a Datadog notebook. It uses the Datadog MCP server or the pup CLI. The folder also holds evaluation data and a judge harness the maintainers use to test the skill.

Capabilities

  • Summarise a single LLM experiment
  • Compare a candidate experiment against a baseline
  • Answer a specific question about experiment results
  • Export the analysis report to a Datadog notebook