← Problem Library
K8s L4 K8S 1340 · 13 min

A fresh cluster passes smoke tests and later loses alerting

The stack installs fine, yet target discovery stays empty because the monitor references a historical release label contract.

K8sPlatform ReliabilityLevel 4Pro13 min
Scenario

A monitoring stack is deployed from a proven internal example and later no targets appear for one newly charted component.

What to check first
  • Identify the primary failure signal in the The Monitoring YAML Inherited a Label Contract the Chart Had Already Abandoned scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Compare live chart labels with the inherited ServiceMonitor selector before editing Prometheus itself.

Questions worth viewing together
What should you verify first when A fresh cluster passes smoke tests and later loses alerting appears?

Community-field Kubernetes problem inspired by OpenInfra and Prometheus community patterns where reused ServiceMonitor selectors expected deprecated Hel... Copied platform examples can carry stale label contracts long after the chart has evolved.

What usually causes A fresh cluster passes smoke tests and later loses alerting in production?

Teams often blame CRD versions when discovery is actually failing on an outdated selector contract.

What should you document after resolving A fresh cluster passes smoke tests and later loses alerting?

Prometheus operator resources should be audited against current chart labels whenever examples are reused across teams.