← Problem Library
K8s L4 K8S 1336 · 13 min

A pod resolves services on old nodes and times out on fresh nodes because CoreDNS was updated but the node local caching DaemonSet still carries the previous upstream service IP

The cluster DNS service is healthy, yet new nodes cache toward an obsolete upstream because one local layer was not rolled with the CoreDNS change.

K8sPlatform ReliabilityLevel 4Pro13 min
Scenario

A DNS service migration completes and later only pods on newly added nodes see timeouts or stale resolution results.

What to check first
  • Identify the primary failure signal in the Cluster DNS Moved, the Node Local Cache Kept the Old Upstream scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Compare node-local cache upstream config with current CoreDNS service identity before replacing CoreDNS pods.

Questions worth viewing together
What should you verify first when A pod resolves services on old nodes and times out on fresh nodes appears?

Community-field Kubernetes problem inspired by Kubernetes community DNS discussions where node-local caches kept an old CoreDNS upstream after migration. Node-local DNS layers can freeze cluster DNS history longer than operators expect.

What usually causes A pod resolves services on old nodes and times out on fresh nodes in production?

Teams often blame CoreDNS performance when the local cache is still talking to the old service IP.

What should you document after resolving A pod resolves services on old nodes and times out on fresh nodes?

DNS service migrations should include every node-local caching layer, not only the core deployment.