Skip to content
Kitbay

agent-observability-online-experiment

MITUpdated on Nov 9, 2026185GitHub stars

A Datadog skill that sets up a live-traffic A/B test for an LLM application instrumented with Agent Observability.

What the skill teaches

A Datadog skill that sets up a live-traffic A/B test for an LLM application instrumented with Agent Observability. After read-only discovery the agent presents one complete plan for approval, then creates a Datadog feature flag for the control and treatment versions, wires the flag into the application code, creates the experiment, attaches an evaluation score as the main metric with token and cost metrics alongside, sets environments and the traffic split, and starts the experiment only if you asked. It needs the Datadog MCP server with its LLM Observability, experiments and feature flag toolsets, and the feature may need enabling for your organisation.

Capabilities

  • Create a Datadog feature flag for two variants
  • Wire the flag into the application
  • Create an online experiment and attach evaluation, token and cost metrics
  • Set environments and traffic allocation
  • Start the experiment on request