← Problem Library
K8s L4 K8S 1416 · 11 min

A Grafana Agent DaemonSet stays healthy and one node never forwards logs

The agent starts fine, yet one node class exposes pod logs under a different symlink chain than the scrape path expects.

K8sPlatform ReliabilityLevel 4Pro11 min
Scenario

A cluster logging rollout works on most nodes and misses logs only from a newly patched worker pool.

What to check first
  • Identify the primary failure signal in the The Agent Mounted the Log Directory and One Node Had Learned a New Symlink Habit scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect actual CRI log symlink targets on each node class before changing relabel rules or log positions.

Questions worth viewing together
What should you verify first when A Grafana Agent DaemonSet stays healthy and one node never forwards logs appears?

Community-field Kubernetes problem inspired by Grafana Agent and logging discussions where one node pool switched to a different CRI log symlink layout. DaemonSet logging issues often come from node-image-specific log path layouts rather than from the agent config alone.

What usually causes A Grafana Agent DaemonSet stays healthy and one node never forwards logs in production?

Teams often blame dropped logs in the pipeline when the mount path never resolved to the active files on one node class.

What should you document after resolving A Grafana Agent DaemonSet stays healthy and one node never forwards logs?

Node image updates should revalidate log symlink layouts wherever hostPath-based collectors are used.