Skip to content
Kitbay

aiml-gpu-training-cluster-investigation

Apache-2.0Updated on Nov 11, 2026138GitHub stars

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS.

What the skill teaches

A skill for AWS DevOps Agent aimed at GPU clusters used for training or inference, whether on SageMaker HyperPod with Slurm or EKS, on AWS ParallelCluster, or on self-managed EC2 and EKS. It checks, node by node and hour by hour, whether kernel Xid logs and HyperPod health detections were actually being collected, so silence is not mistaken for health. It gives each node a replace, reboot or leave-alone verdict against a stated evidence bar, and runs a pre-flight check before long runs covering capacity reservation end times, spare nodes, recovery settings, EFA and idle GPUs. It is read-only.

Capabilities

  • Audit whether GPU error logs were collected per node
  • Decide whether a GPU node should be replaced or rebooted
  • Run a readiness check before a long training job
  • Investigate Xid, ECC, NCCL and NVLink errors