Skip to content
Kitbay
Skill

huggingface-accelerate

MITUpdated on Oct 11, 202614,431GitHub stars

Orchestra Research's skill for Hugging Face Accelerate, the library that runs a PyTorch training loop on several GPUs or machines with few code changes.

What the skill teaches

Orchestra Research's skill for Hugging Face Accelerate, the library that runs a PyTorch training loop on several GPUs or machines with few code changes. It shows the agent how to wrap a model, optimiser and data loader with an Accelerator object, replace the backward call, configure the run interactively and start it with accelerate launch. Workflows cover moving from one GPU to many, mixed precision in FP16, BF16 or FP8, and using DeepSpeed, FSDP or Megatron behind the same interface. References add performance tuning, custom plugins and Megatron integration. It needs the accelerate, torch and transformers packages.

Capabilities

  • Add multi-GPU support to a PyTorch training script
  • Configure mixed precision training
  • Run DeepSpeed, FSDP or Megatron through one API