A networking optimization rollout appears healthy and later one node alone becomes the source of intermittent service traffic loss.
A kube-proxy migration to IPVS passes and one service blackholes traffic
Most nodes converge, yet one path fails because two proxying models overlap on a subset of hosts.
Scenario
What to check first
- Identify the primary failure signal in the The Cluster Migrated to a New Proxy Mode and One Node Still Enforced the Old One scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Audit live proxy mode and residual rules on the affected node before changing Service objects.
Questions worth viewing together
Community-field Kubernetes problem inspired by kube-proxy migration discussions where stale iptables rules persisted after an uneven IPVS rollout. Mode migrations can fail asymmetrically even when the daemonset rollout is mostly green.
Teams often blame CNI or service selector drift when a single node is still enforcing the previous proxy model.
Proxy mode migrations should validate residual state cleanup on every node, not only pod rollout success.
Similar cases seen in the field