← Problem Library
K8s L5 K8S 1318 · 13 min

A DaemonSet upgrade seems fine until one node class loses logs because the hostPath mount moved to a path that only exists on the new image family

Most nodes are healthy, yet a subset stops collecting data because the expected filesystem path differs across node images.

K8sPlatform ReliabilityLevel 5Pro13 min
Scenario

A node image refresh lands and later only some nodes stop yielding host logs while the collector DaemonSet still reports Ready.

What to check first
  • Identify the primary failure signal in the HostPath Contract Changed With the Node Image Family scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Compare host filesystem contracts across node images before changing scrape labels or permissions.

Questions worth viewing together
What should you verify first when A DaemonSet upgrade seems fine until one node class loses logs appears?

Community-field Kubernetes problem inspired by Grafana and node-agent threads where hostPath assumptions broke on one image family. DaemonSet readiness does not prove the mounted host path contains the data you think it does.

What usually causes A DaemonSet upgrade seems fine until one node class loses logs in production?

Teams often blame collector permissions when the path itself no longer exists on one node image line.

What should you document after resolving A DaemonSet upgrade seems fine until one node class loses logs?

HostPath-based agents inherit every filesystem variation across your node fleet.