Skip to content
Kitbay

agent-observability-eval-bootstrap

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill that proposes evaluators for an LLM application by inspecting its production traces in Datadog LLM Observability.

What the skill teaches

A Datadog skill that proposes evaluators for an LLM application by inspecting its production traces in Datadog LLM Observability. Given an ml_app, and optionally the output of a root-cause analysis, the agent samples spans, identifies what should be checked and suggests online LLM-judge evaluators; after you confirm, it creates them in Datadog as disabled drafts. On request it instead writes Python evaluator code for the ddtrace SDK or a JSON specification that is not tied to a framework, and it can also produce a dataset from the traces. It uses the Datadog MCP server or the pup CLI.

Capabilities

  • Sample production LLM traces for an ml_app
  • Propose LLM-judge evaluators from what the traces show
  • Create evaluators in Datadog as disabled drafts after confirmation
  • Generate Python evaluator code or a JSON evaluator spec
  • Emit a dataset from traces