Skip to content
Kitbay

agent-observability-trace-rca

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill for finding out why an LLM application is failing in production.

What the skill teaches

A Datadog skill for finding out why an LLM application is failing in production. Depending on the data available, the agent starts from failing evaluation verdicts, runtime errors or structural oddities in LLM Observability traces, collects the problem spans, groups them and follows each span tree from the symptom back to its cause, then reports the failure patterns it found. Its output can be passed to the eval bootstrap skill. It works through the Datadog MCP server or the pup CLI. The folder also contains evaluation data and a judge harness the maintainers use to test the skill.

Capabilities

  • Find failing spans from eval verdicts or runtime errors
  • Group failures by error type or pattern
  • Trace a failure from symptom to root cause in the span tree
  • Report failure patterns for evaluator generation