← Problem Library
K8s L4 K8S 1391 · 12 min

A metrics pipeline looks healthy and kube-state-metrics dashboards still miss...

Scrapes continue for most targets, yet one monitor no longer discovers the exporter because the selector vocabulary changed with the chart upgrade.

K8sPlatform ReliabilityLevel 4Pro12 min
Scenario

A monitoring chart upgrade lands and later only one dashboard family goes blank despite healthy pods and Services.

What to check first
  • Identify the primary failure signal in the The Exporter Was Running and the Selector Was Looking for Yesterday's Label scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Compare effective labels before tuning queries or recreating the ServiceMonitor.

Questions worth viewing together
What should you verify first when A metrics pipeline looks healthy and kube-state-metrics dashboards still... appears?

Community-field Kubernetes problem inspired by Prometheus and Helm discussions where a ServiceMonitor still matched the old release label. Monitoring regressions after chart upgrades often come from selector vocabulary drift rather than from broken scrapes.

What usually causes A metrics pipeline looks healthy and kube-state-metrics dashboards still... in production?

Teams often blame Prometheus retention or relabeling when discovery simply stopped matching the exporter.

What should you document after resolving A metrics pipeline looks healthy and kube-state-metrics dashboards still...?

Chart label migrations should always be checked against ServiceMonitor selectors and recording rules.