本文へスキップ
Kitbay
スキル

torchtitan

MIT2026年11月11日 更新262,606GitHubスター

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for pretraining large language models with TorchTitan, PyTorch's platform that combines data, tensor, pipeline and context parallelism.

スキルの内容

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for pretraining large language models with TorchTitan, PyTorch's platform that combines data, tensor, pipeline and context parallelism. It covers installing the package, fetching a tokenizer from Hugging Face, choosing named run configurations with command-line overrides, launching an eight-GPU single-node run, writing a SLURM script for multi-node jobs that resume from checkpoints, and enabling Float8 training. For very large models it explains creating a seed checkpoint before splitting work into pipeline stages. Troubleshooting notes cover memory errors and checkpoint resharding, and reference files describe FSDP2, Float8 recipes, checkpoints and adding custom models.

できること

  • Launch single-node TorchTitan pretraining
  • Write multi-node SLURM training jobs
  • Enable Float8 training
  • Combine data, tensor and pipeline parallelism