← Problem Library
K8s L6 K8S 1371 · 15 min

A Cilium upgrade leaves one namespace unreachable because a subset of nodes kept stale endpoint identities after a label normalization changed the identity basis for those pods

Policy objects are updated, yet traffic still drops on some nodes because their identity cache did not converge to the new labels.

K8sPlatform ReliabilityLevel 6Pro15 min
Scenario

A label cleanup is rolled out and later one namespace loses traffic only on a subset of nodes.

What to check first
  • Identify the primary failure signal in the The Labels Changed Cluster-Wide and One Node Still Classified the Pods as Yesterday scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect per-node identity state before broadening policy or recycling the entire namespace.

Questions worth viewing together
What should you verify first when A Cilium upgrade leaves one namespace unreachable appears?

Community-field Kubernetes problem inspired by Cilium and Discuss Kubernetes patterns where stale endpoint identities persisted after label normalization. Identity-aware networking can drift node by node even when policy YAML is identical everywhere.

What usually causes A Cilium upgrade leaves one namespace unreachable in production?

Teams often blame policy syntax when one node is still enforcing the old endpoint identity.

What should you document after resolving A Cilium upgrade leaves one namespace unreachable?

Label migrations in Cilium-backed clusters should include dataplane identity convergence checks.