← Problem Library
K8s L6 K8S 1315 · 16 min

Loki ingestion is healthy and dashboards are empty for one tenant

Logs are stored, yet queries return nothing because write and read paths no longer agree on tenant identity after a migration.

K8sPlatform ReliabilityLevel 6Pro16 min
Scenario

A multi-tenant log stack is renamed or consolidated and only one team's dashboards suddenly appear empty.

What to check first
  • Identify the primary failure signal in the Write Path Moved to the New Tenant, Read Path Did Not scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Verify both query and ingest tenant headers before rebuilding dashboards or retention settings.

Questions worth viewing together
What should you verify first when Loki ingestion is healthy and dashboards are empty for one tenant appears?

Community-field Kubernetes problem inspired by Grafana Community threads where Loki ingest and query paths diverged on tenant identity. Empty dashboards can be a tenant identity mismatch even when ingestion is healthy.

What usually causes Loki ingestion is healthy and dashboards are empty for one tenant in production?

Teams often blame retention or label filters when the query path is simply asking under the wrong tenant.

What should you document after resolving Loki ingestion is healthy and dashboards are empty for one tenant?

Tenant-aware observability stacks require identity alignment on both write and read paths.