← Problem Library
K8s L4 K8S 1334 · 14 min

A node drain hangs even though replicas are healthy because a validating webhook blocks eviction updates after its own CA bundle expired only in one namespace

Workloads are fine, yet drains stop because eviction requests still traverse a webhook path whose trust has silently expired.

K8sPlatform ReliabilityLevel 4Pro14 min
Scenario

A maintenance window begins and a single namespace blocks node drains with admission errors unrelated to application health.

What to check first
  • Identify the primary failure signal in the Maintenance Was Blocked by a Webhook No One Thought Was in the Eviction Path scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Trace eviction admission hooks before forcing drains or lowering PodDisruptionBudgets.

Questions worth viewing together
What should you verify first when A node drain hangs even though replicas are healthy appears?

Community-field Kubernetes problem inspired by Kubernetes operations discussions where expired webhook trust blocked node drain operations. Operational control paths like drain can still depend on app-adjacent admission webhooks.

What usually causes A node drain hangs even though replicas are healthy in production?

Teams often blame PDBs when the real blocker is a failing validating webhook in the eviction path.

What should you document after resolving A node drain hangs even though replicas are healthy?

Webhook certificate hygiene is a maintenance concern, not just an application deployment concern.