← Problem Library
K8s L4 K8S 1352 · 13 min

A Prometheus scrape target stays down because a PodMonitor moved into another namespace and the operator only watches PodMonitors in its original tenant namespace set

The monitor object exists, yet target discovery never includes it because the operator's namespace selector did not follow the new location.

K8sPlatform ReliabilityLevel 4Pro13 min
Scenario

A monitoring refactor centralizes manifests and later one application's metrics vanish despite healthy endpoints and valid PodMonitor syntax.

What to check first
  • Identify the primary failure signal in the The Monitor Moved Somewhere Prometheus Was Never Told to Watch scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Check operator watch scope and namespace selectors before rewriting labels or endpoints.

Questions worth viewing together
What should you verify first when A Prometheus scrape target stays down appears?

Community-field Kubernetes problem inspired by Prometheus community threads where a PodMonitor became invisible after moving outside the operator watch... A valid monitor object can still be invisible if it lives outside the operator's watch boundary.

What usually causes A Prometheus scrape target stays down in production?

Teams often blame bad endpoints when the operator never discovered the monitor in the first place.

What should you document after resolving A Prometheus scrape target stays down?

Namespace moves for monitoring objects should be validated against operator watch selectors, not just manifest correctness.