Skip to content
Kitbay

agent-observability-eval-pipeline

MITUpdated on Nov 9, 2026185GitHub starsIncludes scripts

A Datadog skill that chains the other Agent Observability skills into one guided run for an instrumented LLM application.

What the skill teaches

A Datadog skill that chains the other Agent Observability skills into one guided run for an instrumented LLM application. In stages with a pause for your go-ahead between each, the agent classifies production traces, finds the root causes of failures, proposes evaluators, samples and publishes a dataset to Datadog, generates and runs an experiment, and analyses the results. You can stop after any stage. It ships a small Python helper that publishes the dataset, and it works through the Datadog MCP server or the pup CLI.

Capabilities

  • Classify an LLM application's production traces
  • Root-cause failures and propose evaluators
  • Sample and publish an evaluation dataset to Datadog
  • Generate, run and analyse an experiment
  • Pause for confirmation between stages