Skip to content
Kitbay

agent-observability-experiment-bootstrap

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill that writes a ready-to-run LLM Observability experiment file for your project.

What the skill teaches

A Datadog skill that writes a ready-to-run LLM Observability experiment file for your project. The agent settles the experiment's purpose, dataset (inline records, a local file or a named Datadog dataset), the task under test and the evaluators, then produces a script or notebook using either the Python ddtrace SDK, which is the default, or the Node dd-trace SDK, and checks it locally before reporting. Language-specific guidance and templates sit in its references folder and are loaded only for the chosen SDK. It needs Datadog credentials to record results.

Capabilities

  • Generate an LLM experiment script or notebook
  • Use the Python ddtrace or Node dd-trace SDK
  • Wire in inline, local or Datadog-hosted datasets
  • Add code-based or LLM-judge evaluators