← Problem Library
K8s L5 K8S 1240 · 19 min

Node drain keeps hanging because the CSI node plugin and a strict PDB together prevent the final attachment cleanup from finishing

Maintenance looks blocked by disruption budgets, yet the real deadlock is between attachment cleanup timing and the plugin pods that cannot disappear safely.

K8sPlatform ReliabilityLevel 5Pro19 min
Scenario

A routine node maintenance window stalls when storage-backed workloads and their plugin surfaces cannot be drained in the expected order.

What to check first
  • Identify the primary failure signal in the Drain and CSI Cleanup Deadlock scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Analyze volume attachment cleanup and plugin behavior alongside disruption policy.

Questions worth viewing together
What should you verify first when Node drain keeps hanging appears?

Community-field Kubernetes maintenance problem inspired by operator discussions where CSI cleanup timing and PDBs stalled node drains. A drain stall can be a storage lifecycle problem disguised as a PDB problem.

What usually causes Node drain keeps hanging in production?

Teams often relax the PDB immediately when the deeper issue is attachment cleanup order with CSI components.

What should you document after resolving Node drain keeps hanging?

Storage plugins and disruption policy can deadlock node maintenance in subtle ways.