Scenario
Metrics ingestion resumes after pressure recovery, but dashboards never fill the missing historical gap from the outage window.
Grafana Agent remote_write looks connected but drops a burst of cluster metrics because its WAL sits on ephemeral storage that was rotated away during node pressure recovery
The pipeline reconnects later, yet a whole window of samples is gone because the on-node WAL disappeared during storage cleanup.
What to check first
- Identify the primary failure signal in the Remote Write Recovered but the WAL Vanished First scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Check WAL placement and node pressure cleanup policy before tuning remote_write queues.
Questions worth viewing together
Community-field Kubernetes problem inspired by Grafana Community threads where remote_write recovered but the WAL on ephemeral storage had already been... A recovered remote_write connection does not restore samples if the WAL disappeared first.
Teams often blame backend retention when the missing gap was lost on the sender node.
Telemetry durability depends on where local WAL data lives during node pressure events.
Similar cases seen in the field