Pular para o conteúdo
Kitbay
SkillOficial

huggingface-community-evals

Apache-2.0Atualizado em 11 de nov. de 202611.334Estrelas no GitHubInclui scripts

Hugging Face's skill for evaluating Hub models on the user's own hardware.

O que a skill ensina

Hugging Face's skill for evaluating Hub models on the user's own hardware. It helps the agent choose between inspect-ai and lighteval, and between vLLM, Transformers and accelerate as the inference backend, then run smoke tests and full task runs with bundled uv scripts. One script evaluates a model through inference providers without a local GPU; the others drive local GPU runs. It deliberately stops at producing results: running the same evaluation on Hugging Face Jobs, editing model cards or publishing scores is handed to other skills or workflows. A Hugging Face token is needed for gated models and provider calls.

O que faz

  • Run inspect-ai evaluations against Hub models
  • Run lighteval with vLLM or accelerate on a local GPU
  • Pick an inference backend and fall back when one fails