Skip to content
Kitbay

agent-observability-replay-trace

MITUpdated on Nov 9, 2026185GitHub starsIncludes scripts

A Datadog skill for reworking one LLM Observability trace whose output was wrong.

What the skill teaches

A Datadog skill for reworking one LLM Observability trace whose output was wrong. You give it a trace ID and, optionally, a change to try; the agent fetches the trace through the Datadog MCP server or the pup CLI, finds the entry point in your local code, edits the code if asked, re-runs the application so it emits a new trace, and shows a short diff of old and new output with links to both. You repeat until satisfied. Python apps traced with ddtrace are the main target, and a Python runner template is included for the agent to copy and fill in.

Capabilities

  • Fetch a single LLM Observability trace
  • Re-run the traced request against local code
  • Diff the old and new outputs with links to both traces
  • Loop through edit, replay and compare