Pular para o conteúdo
Kitbay
Skill

distributed-training-deepspeed

MITAtualizado em 11 de nov. de 202633.290Estrelas no GitHub

A skill in the claude-code-templates collection, credited to Orchestra Research, with guidance for training large models across GPUs with Microsoft's DeepSpeed library.

O que a skill ensina

A skill in the claude-code-templates collection, credited to Orchestra Research, with guidance for training large models across GPUs with Microsoft's DeepSpeed library. It says it was generated from DeepSpeed's documentation and covers the ZeRO stages that split optimiser states, gradients and parameters across processes, offloading to CPU memory or NVMe storage, the JSON configuration for batch sizes, optimisers, schedulers and mixed precision, mixture-of-experts models, learning rate range tests, communication-efficient optimisers, pipeline parallelism and the flops profiler. It lists deepspeed, torch, transformers and accelerate as dependencies and notes the DeepSpeed versions some features need. The text is long and mostly condensed documentation.

O que faz

  • Configure DeepSpeed ZeRO stages
  • Offload model state to CPU or NVMe
  • Write a DeepSpeed JSON configuration
  • Set up mixture-of-experts training