本文へスキップ
Kitbay
スキル

tensorrt-llm

MIT2026年11月11日 更新262,606GitHubスター

An optional Hermes Agent skill, credited to Orchestra Research, on serving large language models with NVIDIA's TensorRT-LLM library on NVIDIA GPUs under Linux.

スキルの内容

An optional Hermes Agent skill, credited to Orchestra Research, on serving large language models with NVIDIA's TensorRT-LLM library on NVIDIA GPUs under Linux. It says when to choose it over vLLM or llama.cpp, how to install it from NGC Docker images or pip with the required CUDA, TensorRT and Python versions, and how to use the Python LLM API, the trtllm-serve server and its OpenAI-style chat endpoint. Patterns cover FP8 and other quantisation, tensor parallelism across GPUs and in-flight batching, with listed model families and features such as speculative decoding and LoRA serving. Three reference files hold optimisation, multi-GPU and production serving notes. Its throughput figures are the library's own claims.

できること

  • Install TensorRT-LLM from NGC or pip
  • Serve a model with trtllm-serve
  • Load quantised models in FP8
  • Spread a model across several GPUs