← Problem Library
K8s L4 K8S 1332 · 13 min

A ServiceMonitor appears valid and Prometheus still scrapes nothing

The object exists, yet no targets are discovered because the namespace watch scope no longer includes the new monitor location.

K8sPlatform ReliabilityLevel 4Pro13 min
Scenario

A monitoring refactor centralizes manifests and later one application silently disappears from scrape targets.

What to check first
  • Identify the primary failure signal in the The Monitor Moved Outside the Part of the Cluster Prometheus Was Watching scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Check namespace selectors and discovery scope before rewriting relabel configs.

Questions worth viewing together
What should you verify first when A ServiceMonitor appears valid and Prometheus still scrapes nothing appears?

Community-field Kubernetes problem inspired by Prometheus community threads where a valid ServiceMonitor was invisible due to namespace selector scope. Prometheus can miss perfectly valid monitor objects if they moved outside the namespaces it watches.

What usually causes A ServiceMonitor appears valid and Prometheus still scrapes nothing in production?

Teams often blame broken endpoints when discovery never included the ServiceMonitor at all.

What should you document after resolving A ServiceMonitor appears valid and Prometheus still scrapes nothing?

Namespace refactors should be tested against every selector in the monitoring stack.