Pular para o conteúdo
Kitbay
SkillOficial

aiml-gpu-training-cluster-investigation

Apache-2.0Atualizado em 19 de nov. de 2026143Estrelas no GitHub

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS.

O que a skill ensina

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS. It checks, node by node and hour by hour, whether kernel Xid logs and HyperPod health detections were actually being collected, so silence is not mistaken for health. It gives each node a replace, reboot or leave-alone verdict against a stated evidence bar, and runs a pre-flight check before long runs covering capacity reservation end times, spare nodes, recovery settings, EFA and idle GPUs. It is read-only.

O que faz

  • Audit whether GPU error logs were collected per node
  • Decide whether a GPU node should be replaced or rebooted
  • Run a readiness check before a long training job
  • Investigate Xid, ECC, NCCL and NVLink errors