Scenario
A cluster networking optimization is rolled out and later only one node intermittently drops service traffic even though the migration reported success.
A kube-proxy migration to IPVS mostly works and one node blackholes service traffic because stale iptables rules from the old mode were never cleaned up on that host
The daemonset is healthy, yet service traffic fails through one node because it still carries residual rules from the previous proxy mode.
What to check first
- Identify the primary failure signal in the The Cluster Changed Proxy Mode and One Node Kept the Old Rules Underneath scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Audit the failing node's effective ruleset before changing Services or CNI configuration.
Questions worth viewing together
Community-field Kubernetes problem inspired by kube-proxy migration discussions where one node retained stale iptables rules after moving to IPVS. Mode migrations can fail asymmetrically on one host even when the rollout looks healthy cluster-wide.
Teams often blame CNI or service selectors when a single node still enforces the previous proxy rules.
Proxy mode changes should include residual-state checks on every node, not only daemonset rollout health.
Similar cases seen in the field