Typical symptom
When GPU nodes exist but the workload keeps failing to schedule is rarely a single wrong setting — it usually appears when the deploy boundary, runtime state, cache, permissions, and network path drift together. Users arriving from search should first narrow the symptom, using this guide's search intent — Targets search intents like GPU node present but workload not scheduled, Pod pending on GPU node, and taint toleration GPU mismatch. — as the basis for the first signals to check.
Early on, rather than attempting a full rollback or a blind restart, check which node, Pod, job, user, or path the blast radius is tied to. If the scope is narrow, compare recent changes against the healthy state; if it is wide, start from the shared dependencies.
Signals to check first
The first thing to look at is not the last error line but the boundary where the same failure recurs. Grouping the problem around signals like insufficient nvidia.com/gpu, taint mismatch, device plugin not ready, node selector excludes gpu node narrows the root-cause candidates even when the logs are long.
- First compare the Pod's GPU request against the node's actual allocatable value.
- Check that taint, toleration, nodeSelector, and affinity don't conflict at the same time.
- Check whether the device plugin and driver daemonset are actually Ready.
- Check whether the GPU node is cordoned or the autoscaler is imposing another constraint.
- Directly compare the spec against a healthy GPU workload in the same namespace.
Logs and CLI examples
The commands below don't hand you the answer directly; they are the first observation points for narrowing the cause. Comparing their output against a known-good point in time or a healthy resource with the same role cuts down time spent just retrying.
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previousCommon misdiagnoses
The most dangerous pattern in operational incidents is mistaking the symptom name for the cause. The same timeout, permission denied, or rollout failure can have its real cause in a different layer — cache, permission inheritance, Secret scope, stale client connections, or proxy headers.
- Assuming that if the GPU node is visible, the scheduler can pick it right away.
- Suspecting only the autoscaler without checking taint/toleration and device plugin state.
Safe recovery order
Recovery starts at the smallest unit. First pin the current state with read-only checks, then verify changes on a limited-impact resource. Hard-to-reverse actions like a full service restart, clearing the entire cache, or relaxing security policy should be chosen only after the root-cause candidates are narrowed.
- GPU issues often look like a resource shortage but are actually a node policy conflict.
- If the driver and device plugin aren't Ready, the scheduler won't count the node's GPUs as usable even though the node is visible.
Prevention
After an incident ends, record "why that state lingered" rather than just a one-line cause. Check whether there was a gap between automation and operational procedure — in the deploy pipeline, runtime reload, permission inheritance, certificate renewal, or network policy.
Following the related hubs and problems lets you re-diagnose the same symptom in other environments.
Related InfraTree problems
The problems below are public exercises for practicing this guide as real scenarios. Solving them after reading lets you practice splitting signals first and writing out the recovery direction as sentences.