Skip to content
Kitbay

hf-cloud-serving-image-selection

Apache-2.0Updated on Nov 11, 202611,334GitHub starsIncludes scripts

A Hugging Face skill for choosing the serving container when deploying a model to a SageMaker endpoint.

What the skill teaches

A Hugging Face skill for choosing the serving container when deploying a model to a SageMaker endpoint. It requires a Hugging Face container family (vLLM, vLLM-Omni, SGLang, TEI or PyTorch inference) whenever one can serve the model, gives the few reasons for falling back to a generic image, and reads image URIs from AWS's Deep Learning Containers catalog. It covers choosing between TEI and vLLM for rerankers, AMI and CUDA requirements, container settings, private-network download problems, images known to be broken and first-boot download time. A bundled script mirrors images when a region lacks one.

Capabilities

  • Pick the serving container for a model on SageMaker
  • Look up current image URIs from AWS's container catalog
  • Configure vLLM and TEI containers
  • Mirror an image to another region