agent-observability-auto-experiment
A Datadog skill that runs a measured improvement loop on an LLM application's code using its Datadog LLM Observability data.
What the skill teaches
A Datadog skill that runs a measured improvement loop on an LLM application's code using its Datadog LLM Observability data. The agent loads an evaluation corpus (from an ml_app, a dataset, an annotation queue, a set of trace IDs or a local file) into Datadog Datasets, builds a local scoring harness, records a baseline score, then makes one change at a time and keeps it only when the re-scored result improves. It works through the Datadog MCP server or the pup CLI, asks for its inputs before starting and estimates run cost from the measured baseline. Harness templates in Python and Node are included.
Capabilities
- Load traces, datasets or annotation queues into Datadog Datasets
- Build a local evaluation harness and score a baseline
- Make one focused code or prompt change per iteration and re-score it
- Keep changes only when the score improves
- Work through the Datadog MCP server or the pup CLI
About
- Publisher
- Datadog
- Tool
- Datadog
Install
What's inside
- references/eval_harness_template.mjs
- references/eval_harness_template.py
- references/rubrics.md
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Datadog | |
| Publisher | Datadog | source |
| Official status | Officialdatadog-labs org; MIT licence copyright Datadog | source |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | No | source |
| Platforms | Not stated in its documentation | |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Nov 9, 2026185 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository |
More from Datadog
2 servers · 70 skills
- agent-observability-build-eval-from-annotations (Skill)
- agent-observability-eval-bootstrap (Skill)
- agent-observability-eval-pipeline (Skill)
- agent-observability-experiment-analyzer (Skill)
- agent-observability-experiment-bootstrap (Skill)
- agent-observability-online-experiment (Skill)
Badge for your README
Maintain this project? Add a badge that links to this page.
[](https://kitbay.app/skills/datadog-labs-agent-observability-auto-experiment/)
Sources
- Skill file (SKILL.md)github.com/datadog-labs/agent-skillsNov 10, 2026
- README (README.md)github.com/datadog-labs/agent-skillsNov 13, 2026
- Licence file (LICENSE)github.com/datadog-labs/agent-skills · Backs: Licence, Official statusNov 13, 2026
- GitHub repositorygithub.com/datadog-labs/agent-skills · Backs: Publisher, Activity, LicenceNov 11, 2026
- Folder in the repository (agent-observability-auto-experiment)github.com/datadog-labs/agent-skills · Backs: Includes scriptsNov 10, 2026
Other MCP servers and skills for Datadog
- agent-observability-build-eval-from-annotations(Skill)
- agent-observability-eval-bootstrap(Skill)
- agent-observability-eval-pipeline(Skill)
- agent-observability-experiment-analyzer(Skill)
- agent-observability-experiment-bootstrap(Skill)
- agent-observability-online-experiment(Skill)
- datadog-app(Skill)
- dd-pup(Skill)
- View all →