← Problem Library
K8s L4 K8S 1356 · 12 min

A Grafana dashboard for kube-state-metrics goes blank because the relabel config now drops owner_kind while the dashboard variables still join on it for higher-level workloads

Scrapes are healthy, yet workload panels are empty because the join key used by the dashboard was stripped out during relabel cleanup.

K8sPlatform ReliabilityLevel 4Pro12 min
Scenario

A label normalization cleanup lands and later Kubernetes workload dashboards lose hierarchy views even though the source targets remain up.

What to check first
  • Identify the primary failure signal in the The Metric Survived and the Label the Dashboard Needed Did Not scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect the final metric label set before changing the dashboard queries or scrape intervals.

Questions worth viewing together
What should you verify first when A Grafana dashboard for kube-state-metrics goes blank appears?

Community-field Kubernetes problem inspired by Grafana and Prometheus community patterns where relabeling removed owner_kind and blanked workload dashbo... Dashboard emptiness can be a post-scrape label problem even when collection is perfectly healthy.

What usually causes A Grafana dashboard for kube-state-metrics goes blank in production?

Teams often blame kube-state-metrics instability when the relabel stage removed the ownership label dashboards joined on.

What should you document after resolving A Grafana dashboard for kube-state-metrics goes blank?

Relabel cleanup should be regression-tested against dashboard joins and template variables, not only ingestion volume.