本文へスキップ
Kitbay
スキル

evaluating-llms-harness

MIT2026年11月11日 更新262,606GitHubスター

An optional skill in the Hermes Agent repository, credited to Orchestra Research, for benchmarking language models with EleutherAI's lm-evaluation-harness.

スキルの内容

An optional skill in the Hermes Agent repository, credited to Orchestra Research, for benchmarking language models with EleutherAI's lm-evaluation-harness. The agent installs the lm-eval package and runs academic benchmarks such as MMLU, GSM8K, HellaSwag, ARC and HumanEval against Hugging Face checkpoints, quantised models or local checkpoints, optionally with vLLM as a faster backend. Worked examples show how to evaluate checkpoints during training and plot a learning curve, compare several models in a Markdown table, and deal with slow runs or memory errors. Code benchmarks need an explicit flag because they run generated code. It targets Linux and macOS.

できること

  • Run MMLU, GSM8K and other benchmarks on a model
  • Evaluate checkpoints during training
  • Compare several models in one table
  • Use vLLM as a faster evaluation backend