Skip to content
Kitbay

huggingface-community-evals

Apache-2.0Updated on Nov 11, 202611,334GitHub starsIncludes scripts

Hugging Face's skill for evaluating Hub models on the user's own hardware.

What the skill teaches

Hugging Face's skill for evaluating Hub models on the user's own hardware. It helps the agent choose between inspect-ai and lighteval, and between vLLM, Transformers and accelerate as the inference backend, then run smoke tests and full task runs with bundled uv scripts. One script evaluates a model through inference providers without a local GPU; the others drive local GPU runs. It deliberately stops at producing results: running the same evaluation on Hugging Face Jobs, editing model cards or publishing scores is handed to other skills or workflows. A Hugging Face token is needed for gated models and provider calls.

Capabilities

  • Run inspect-ai evaluations against Hub models
  • Run lighteval with vLLM or accelerate on a local GPU
  • Pick an inference backend and fall back when one fails