agent-observability-build-eval-from-annotations
A Datadog skill that turns human labels in an LLM Observability annotation queue into an evaluator.
What the skill teaches
A Datadog skill that turns human labels in an LLM Observability annotation queue into an evaluator. The agent reads the queue, finds which part of each trace the labels refer to, drafts an LLM judge, agrees a scoring metric with you and measures the judge against the labelled rows. It then revises the judge one change at a time for a limited number of rounds, checks it on held-out rows and creates the result in Datadog as an evaluator that stays switched off until you enable it. It bundles a local judge runner and a scoring script and can use the Datadog MCP server or the pup CLI.
Capabilities
- Read labelled interactions from an annotation queue
- Draft an LLM judge that predicts the human labels
- Score the judge against existing labels with an agreed metric
- Refine the judge over several measured rounds
- Create the final evaluator in Datadog in a disabled state
About
- Publisher
- Datadog
- Tool
- Datadog
Install
What's inside
- references/judge_runner.py
- references/rubrics.md
- references/scoring.py
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
This skill includes scripts that your agent may run.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Datadog | |
| Publisher | Datadog | source |
| Official status | Officialdatadog-labs org; MIT licence copyright Datadog | source |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | Yes | source |
| Platforms | Not stated in its documentation | |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Nov 9, 2026185 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository |
More from Datadog
2 servers · 70 skills
- agent-observability-auto-experiment (Skill)
- agent-observability-eval-bootstrap (Skill)
- agent-observability-eval-pipeline (Skill)
- agent-observability-experiment-analyzer (Skill)
- agent-observability-experiment-bootstrap (Skill)
- agent-observability-online-experiment (Skill)
Badge for your README
Maintain this project? Add a badge that links to this page.
[](https://kitbay.app/skills/datadog-labs-agent-observability-build-eval-from-annotations/)
Sources
- Skill file (SKILL.md)github.com/datadog-labs/agent-skillsNov 10, 2026
- README (README.md)github.com/datadog-labs/agent-skillsNov 13, 2026
- Licence file (LICENSE)github.com/datadog-labs/agent-skills · Backs: Licence, Official statusNov 13, 2026
- GitHub repositorygithub.com/datadog-labs/agent-skills · Backs: Publisher, Activity, LicenceNov 11, 2026
- File in the repository (agent-observability-build-eval-from-annotations)github.com/datadog-labs/agent-skills · Backs: Includes scriptsNov 13, 2026
- Folder in the repository (agent-observability-build-eval-from-annotations)github.com/datadog-labs/agent-skills · Backs: Includes scriptsNov 10, 2026
Other MCP servers and skills for Datadog
- agent-observability-auto-experiment(Skill)
- agent-observability-eval-bootstrap(Skill)
- agent-observability-eval-pipeline(Skill)
- agent-observability-experiment-analyzer(Skill)
- agent-observability-experiment-bootstrap(Skill)
- agent-observability-online-experiment(Skill)
- datadog-app(Skill)
- dd-pup(Skill)
- View all →