agent-observability-online-experiment
A Datadog skill that sets up a live-traffic A/B test for an LLM application instrumented with Agent Observability.
What the skill teaches
A Datadog skill that sets up a live-traffic A/B test for an LLM application instrumented with Agent Observability. After read-only discovery the agent presents one complete plan for approval, then creates a Datadog feature flag for the control and treatment versions, wires the flag into the application code, creates the experiment, attaches an evaluation score as the main metric with token and cost metrics alongside, sets environments and the traffic split, and starts the experiment only if you asked. It needs the Datadog MCP server with its LLM Observability, experiments and feature flag toolsets, and the feature may need enabling for your organisation.
Capabilities
- Create a Datadog feature flag for two variants
- Wire the flag into the application
- Create an online experiment and attach evaluation, token and cost metrics
- Set environments and traffic allocation
- Start the experiment on request
About
- Publisher
- Datadog
- Tool
- Datadog
Install
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Datadog | |
| Publisher | Datadog | source |
| Official status | Officialdatadog-labs org; MIT licence copyright Datadog | source |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | No | source |
| Platforms | Not stated in its documentation | |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Nov 9, 2026185 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository |
More from Datadog
2 servers · 70 skills
- agent-observability-auto-experiment (Skill)
- agent-observability-build-eval-from-annotations (Skill)
- agent-observability-eval-bootstrap (Skill)
- agent-observability-eval-pipeline (Skill)
- agent-observability-experiment-analyzer (Skill)
- agent-observability-experiment-bootstrap (Skill)
Badge for your README
Maintain this project? Add a badge that links to this page.
[](https://kitbay.app/skills/datadog-labs-agent-observability-online-experiment/)
Sources
- Skill file (SKILL.md)github.com/datadog-labs/agent-skillsNov 10, 2026
- README (README.md)github.com/datadog-labs/agent-skillsNov 13, 2026
- Licence file (LICENSE)github.com/datadog-labs/agent-skills · Backs: Licence, Official statusNov 13, 2026
- GitHub repositorygithub.com/datadog-labs/agent-skills · Backs: Publisher, Activity, LicenceNov 11, 2026
- File in the repository (agent-observability-online-experiment)github.com/datadog-labs/agent-skills · Backs: Includes scriptsNov 13, 2026
- Folder in the repository (agent-observability-online-experiment)github.com/datadog-labs/agent-skills · Backs: Includes scriptsNov 10, 2026
Other MCP servers and skills for Datadog
- agent-observability-auto-experiment(Skill)
- agent-observability-build-eval-from-annotations(Skill)
- agent-observability-eval-bootstrap(Skill)
- agent-observability-eval-pipeline(Skill)
- agent-observability-experiment-analyzer(Skill)
- agent-observability-experiment-bootstrap(Skill)
- datadog-app(Skill)
- dd-pup(Skill)
- View all →