Pular para o conteúdo
Kitbay
Skill

serving-llms-vllm

MITAtualizado em 11 de nov. de 2026262.606Estrelas no GitHub

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM.

O que a skill ensina

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for serving large language models with vLLM. It covers three workflows: running an OpenAI-compatible API server with settings for different model sizes, load testing, Prometheus monitoring and the vllm-openai Docker image; offline batch generation from a prompt file to a JSONL results file; and serving AWQ, GPTQ or FP8 quantised models on limited GPU memory. It gives GPU sizing per model size, explains when other runtimes such as llama.cpp or TensorRT-LLM fit better, and lists fixes for out-of-memory errors, slow first tokens and low throughput. Four reference files go deeper on deployment, tuning, quantisation and troubleshooting.

O que faz

  • Start an OpenAI-compatible vLLM server
  • Run offline batch generation
  • Serve quantised models on smaller GPUs
  • Troubleshoot memory and throughput problems