Zum Inhalt springen
Kitbay

hf-cloud-serving-image-selection

Apache-2.0Aktualisiert am 11. Nov. 202611.334GitHub-SterneEnthält Skripte

A Hugging Face skill for choosing the serving container when deploying a model to a SageMaker endpoint.

Was der Skill vermittelt

A Hugging Face skill for choosing the serving container when deploying a model to a SageMaker endpoint. It requires a Hugging Face container family (vLLM, vLLM-Omni, SGLang, TEI or PyTorch inference) whenever one can serve the model, gives the few reasons for falling back to a generic image, and reads image URIs from AWS's Deep Learning Containers catalog. It covers choosing between TEI and vLLM for rerankers, AMI and CUDA requirements, container settings, private-network download problems, images known to be broken and first-boot download time. A bundled script mirrors images when a region lacks one.

Funktionen

  • Pick the serving container for a model on SageMaker
  • Look up current image URIs from AWS's container catalog
  • Configure vLLM and TEI containers
  • Mirror an image to another region