← Problem Library
K8s L5 K8S 1306 · 14 min

Promtail tails files successfully until log rotation accelerates because the CRI symlink target changes faster than the positions file can catch up

Short-lived containers emit logs, but some lines vanish during fast rotation because the watched path and the tracked inode move out of sync.

K8sPlatform ReliabilityLevel 5Pro14 min
Scenario

A noisy cluster starts losing short-lived container logs even though Promtail itself reports no crash or auth error.

What to check first
  • Identify the primary failure signal in the Rotation Moved the Real File Faster Than Positions Followed scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect CRI symlink behavior and positions file semantics before changing scrape labels.

Questions worth viewing together
What should you verify first when Promtail tails files successfully until log rotation accelerates appears?

Community-field Kubernetes problem inspired by Grafana Community discussions where Promtail lost lines during fast CRI log rotation. Log loss during rotation is often about inode tracking, not about collector connectivity.

What usually causes Promtail tails files successfully until log rotation accelerates in production?

Teams often blame Loki ingestion when Promtail never followed the rotated file edge correctly.

What should you document after resolving Promtail tails files successfully until log rotation accelerates?

CRI log symlink handling under high churn can differ from ordinary file tail expectations.