← Problem Library
K8s L6 K8S 1331 · 16 min

A Cilium policy rollout looks correct and one namespace still loses traffic because the endpoint identity cache on a subset of nodes never refreshed after a label normalization

Policies are present, yet traffic drops because some nodes still enforce an older endpoint identity view tied to pre-normalization labels.

K8sPlatform ReliabilityLevel 6Pro16 min
Scenario

A label cleanup is followed by namespace-specific traffic loss that appears only on a subset of nodes.

What to check first
  • Identify the primary failure signal in the The Policy Was New, the Node Identity Cache Was Not scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Compare endpoint identity and policy resolution per node before broadening the allow rules.

Questions worth viewing together
What should you verify first when A Cilium policy rollout looks correct and one namespace still loses traffic appears?

Community-field Kubernetes problem inspired by Cilium and Kubernetes community patterns where identity caches lagged behind label normalization. Policy truth can drift node by node when identity caches lag behind label change events.

What usually causes A Cilium policy rollout looks correct and one namespace still loses traffic in production?

Teams often blame the policy YAML when the node is still classifying endpoints under the old identity.

What should you document after resolving A Cilium policy rollout looks correct and one namespace still loses traffic?

Identity-based policy engines need explicit cache convergence checks after widespread relabeling.