Skip to content
Kitbay
Skill

torchtitan

MITUpdated on Nov 11, 2026262,606GitHub stars

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for pretraining large language models with TorchTitan, PyTorch's platform that combines data, tensor, pipeline and context parallelism.

What the skill teaches

An optional skill in Nous Research's Hermes Agent, credited to Orchestra Research, for pretraining large language models with TorchTitan, PyTorch's platform that combines data, tensor, pipeline and context parallelism. It covers installing the package, fetching a tokenizer from Hugging Face, choosing named run configurations with command-line overrides, launching an eight-GPU single-node run, writing a SLURM script for multi-node jobs that resume from checkpoints, and enabling Float8 training. For very large models it explains creating a seed checkpoint before splitting work into pipeline stages. Troubleshooting notes cover memory errors and checkpoint resharding, and reference files describe FSDP2, Float8 recipes, checkpoints and adding custom models.

Capabilities

  • Launch single-node TorchTitan pretraining
  • Write multi-node SLURM training jobs
  • Enable Float8 training
  • Combine data, tensor and pipeline parallelism