evaluating-llms-harness
An optional skill in the Hermes Agent repository, credited to Orchestra Research, for benchmarking language models with EleutherAI's lm-evaluation-harness.
What the skill teaches
An optional skill in the Hermes Agent repository, credited to Orchestra Research, for benchmarking language models with EleutherAI's lm-evaluation-harness. The agent installs the lm-eval package and runs academic benchmarks such as MMLU, GSM8K, HellaSwag, ARC and HumanEval against Hugging Face checkpoints, quantised models or local checkpoints, optionally with vLLM as a faster backend. Worked examples show how to evaluate checkpoints during training and plot a learning curve, compare several models in a Markdown table, and deal with slow runs or memory errors. Code benchmarks need an explicit flag because they run generated code. It targets Linux and macOS.
Capabilities
- Run MMLU, GSM8K and other benchmarks on a model
- Evaluate checkpoints during training
- Compare several models in one table
- Use vLLM as a faster evaluation backend
About
- Publisher
- Nous Research
- Tool
- Hugging Face
Install
The project doesn't document install steps that we could confirm. See the project's own instructions
What's inside
- references/api-evaluation.md
- references/benchmark-guide.md
- references/custom-tasks.md
- references/distributed-eval.md
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Hugging Face | |
| Publisher | Nous Research | source |
| Official status | Community (not published by the tool's vendor) | source |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | No | source |
| Platforms | Linux, macOS | source |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Nov 11, 2026262,606 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository Website |
Sources
- Skill file (SKILL.md)github.com/nousresearch/hermes-agent · Backs: Licence, Official status, PlatformsNov 16, 2026
- GitHub repositorygithub.com/nousresearch/hermes-agent · Backs: Publisher, Activity, LicenceNov 11, 2026
- GitHub API recordgithub.com/nousresearch/hermes-agent · Backs: Includes scriptsNov 16, 2026
- Skill file (SKILL.md)github.com/NousResearch/hermes-agentNov 10, 2026
- Folder in the repository (evaluating-llms-harness)github.com/NousResearch/hermes-agent · Backs: Includes scriptsNov 10, 2026