Skip to content
Kitbay

creating-mcp-agent-evals

MITUpdated on Nov 10, 202615,564GitHub stars

A maintainer skill in Apify's MCP server repository for building Langfuse evaluation suites that test how well agents use one family of the server's tools, such as storage or runs.

What the skill teaches

A maintainer skill in Apify's MCP server repository for building Langfuse evaluation suites that test how well agents use one family of the server's tools, such as storage or runs. It sets out a process: list every tool and argument group, check real platform behaviour with small scripts before writing cases, write easy, medium and hard cases in rounds, calibrate on a strong model and then test weaker ones, and fix tool outputs before descriptions. A diagnosis table helps decide whether a failing case, the judge, the product, the tool text or the model is at fault. A reference file holds commands and patterns.

Capabilities

  • Plan eval cases for an Apify MCP tool family
  • Calibrate evals across stronger and weaker models
  • Diagnose why an eval case fails
  • Improve tool outputs based on eval results