本文へスキップ
Kitbay
スキル

inference-optimisation

MIT2026年11月16日 更新198GitHubスタースクリプトあり

A skill from Sybil Solutions for making a local large language model server faster, covering decode and prefill speed, speculative decoding, KV cache size, startup and time to first token on any hardware and quantisation.

スキルの内容

A skill from Sybil Solutions for making a local large language model server faster, covering decode and prefill speed, speculative decoding, KV cache size, startup and time to first token on any hardware and quantisation. The agent works out theoretical ceilings from measured memory bandwidth and compute first, then changes one setting at a time against interleaved control runs, and every result must pass a long-context stability check. Bundled Python and shell scripts benchmark any OpenAI-compatible server, compute the ceilings, queue runs one after another and report a container's real memory use. Notes cover engines such as vLLM, SGLang, llama.cpp and ExLlama.

できること

  • Benchmark an OpenAI-compatible model server
  • Compute theoretical speed ceilings
  • Tune one serving setting at a time
  • Check long-context stability after each change