Skip to content
Kitbay

agent-observability-session-classify

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill that judges whether a user got what they wanted from an AI assistant interaction recorded in Datadog.

What the skill teaches

A Datadog skill that judges whether a user got what they wanted from an AI assistant interaction recorded in Datadog. It works on a single assistant session (adding Real User Monitoring and audit-trail signals), a single Agent Observability trace, or a sample of sessions or traces from one LLM application. The agent reads the conversation and tool results and returns a short verdict with a one-line reason by default. Its output can feed the trace root-cause and eval pipeline skills. It uses the Datadog MCP server or the pup CLI.

Capabilities

  • Classify a single session or trace as satisfied or not
  • Combine trace content with RUM behaviour signals
  • Sample and classify many sessions from one LLM app
  • Produce labels for root-cause analysis and evaluation