本文へスキップ
Kitbay
スキル公式

creating-mcp-agent-evals

MIT2026年11月10日 更新15,564GitHubスター

A maintainer skill in Apify's MCP server repository for building Langfuse evaluation suites that test how well agents use one family of the server's tools, such as storage or runs.

スキルの内容

A maintainer skill in Apify's MCP server repository for building Langfuse evaluation suites that test how well agents use one family of the server's tools, such as storage or runs. It sets out a process: list every tool and argument group, check real platform behaviour with small scripts before writing cases, write easy, medium and hard cases in rounds, calibrate on a strong model and then test weaker ones, and fix tool outputs before descriptions. A diagnosis table helps decide whether a failing case, the judge, the product, the tool text or the model is at fault. A reference file holds commands and patterns.

できること

  • Plan eval cases for an Apify MCP tool family
  • Calibrate evals across stronger and weaker models
  • Diagnose why an eval case fails
  • Improve tool outputs based on eval results