serving-llms-vllm
An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM.
What the skill teaches
An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM. It covers three workflows: running an OpenAI-compatible API server with settings for different model sizes, load testing, Prometheus monitoring and the vllm-openai Docker image; offline batch generation from a prompt file to a JSONL results file; and serving AWQ, GPTQ or FP8 quantised models on limited GPU memory. It gives GPU sizing per model size, explains when other runtimes such as llama.cpp or TensorRT-LLM fit better, and lists fixes for out-of-memory errors, slow first tokens and low throughput. Four reference files go deeper on deployment, tuning, quantisation and troubleshooting.
Capabilities
- Start an OpenAI-compatible vLLM server
- Run offline batch generation
- Serve quantised models on smaller GPUs
- Troubleshoot memory and throughput problems
About
- Publisher
- Nous Research
Install
The project doesn't document install steps that we could confirm. See the project's own instructions
What's inside
- references/optimization.md
- references/quantization.md
- references/server-deployment.md
- references/troubleshooting.md
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Not stated in its documentation | |
| Publisher | Nous Research | source |
| Official status | Not stated in its documentation | |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | No | source |
| Platforms | Linux, macOS | source |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Nov 11, 2026262,606 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository Website |
Other copies of this skill (1)
These differ from this skill's text. They may be adapted copies or older versions.
- serving-llms-vllmlord1egypt/ra-skills
More from Nous Research
42 skills
- inference-sh-cli (Skill)
- comfyui (Skill)
- excalidraw (Skill)
- unreal-mcp (Skill)
- jupyter-notebook (Skill)
- docker-management (Skill)
Sources
- Skill file (SKILL.md)github.com/nousresearch/hermes-agent · Backs: Licence, Platforms, Includes scriptsNov 16, 2026
- GitHub repositorygithub.com/nousresearch/hermes-agent · Backs: Publisher, Activity, LicenceNov 11, 2026
- Skill file (SKILL.md)github.com/NousResearch/hermes-agentNov 10, 2026
- Folder in the repository (serving-llms-vllm)github.com/NousResearch/hermes-agent · Backs: Includes scriptsNov 10, 2026