本文へスキップ
Kitbay
スキル

serving-llms-vllm

MIT2026年11月11日 更新262,606GitHubスター

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM.

スキルの内容

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM. It covers three workflows: running an OpenAI-compatible API server with settings for different model sizes, load testing, Prometheus monitoring and the vllm-openai Docker image; offline batch generation from a prompt file to a JSONL results file; and serving AWQ, GPTQ or FP8 quantised models on limited GPU memory. It gives GPU sizing per model size, explains when other runtimes such as llama.cpp or TensorRT-LLM fit better, and lists fixes for out-of-memory errors, slow first tokens and low throughput. Four reference files go deeper on deployment, tuning, quantisation and troubleshooting.

できること

  • Start an OpenAI-compatible vLLM server
  • Run offline batch generation
  • Serve quantised models on smaller GPUs
  • Troubleshoot memory and throughput problems