本文へスキップ
Kitbay
スキル公式

aiml-gpu-training-cluster-investigation

Apache-2.02026年11月11日 更新138GitHubスター

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS.

スキルの内容

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS. It checks, node by node and hour by hour, whether kernel Xid logs and HyperPod health detections were actually being collected, so silence is not mistaken for health. It gives each node a replace, reboot or leave-alone verdict against a stated evidence bar, and runs a pre-flight check before long runs covering capacity reservation end times, spare nodes, recovery settings, EFA and idle GPUs. It is read-only.

できること

  • Audit whether GPU error logs were collected per node
  • Decide whether a GPU node should be replaced or rebooted
  • Run a readiness check before a long training job
  • Investigate Xid, ECC, NCCL and NVLink errors