Skip to content
Kitbay
Skill

serving-llms-vllm

MITUpdated on Nov 11, 2026262,606GitHub stars

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM.

What the skill teaches

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM. It covers three workflows: running an OpenAI-compatible API server with settings for different model sizes, load testing, Prometheus monitoring and the vllm-openai Docker image; offline batch generation from a prompt file to a JSONL results file; and serving AWQ, GPTQ or FP8 quantised models on limited GPU memory. It gives GPU sizing per model size, explains when other runtimes such as llama.cpp or TensorRT-LLM fit better, and lists fixes for out-of-memory errors, slow first tokens and low throughput. Four reference files go deeper on deployment, tuning, quantisation and troubleshooting.

Capabilities

  • Start an OpenAI-compatible vLLM server
  • Run offline batch generation
  • Serve quantised models on smaller GPUs
  • Troubleshoot memory and throughput problems