← Problem Library
Security L5 SECURITY-1358 · 12 min

A Prometheus security alert should route to the security team and lands on...

The alert fires, yet paging goes to the wrong place because the ownership label vanished earlier in the pipeline.

SecurityPlatform ReliabilityLevel 5Pro12 min
Scenario

An alerting cleanup is deployed and later security incidents page the default on-call instead of the intended responders.

What to check first
  • Identify the primary failure signal in the The Alert Stayed Intact and the Label That Said Who Owned It Did Not scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Trace label state into Alertmanager before editing routes or receiver names.

Questions worth viewing together
What should you verify first when A Prometheus security alert should route to the security team and lands on... appears?

Community-field security problem inspired by Prometheus and Grafana threads where relabeling stripped ownership labels before Alertmanager routing. Alert routing failures often start upstream in relabel logic, not inside Alertmanager's match tree.

What usually causes A Prometheus security alert should route to the security team and lands on... in production?

Teams often blame Alertmanager matching when the decisive label already disappeared before routing.

What should you document after resolving A Prometheus security alert should route to the security team and lands on...?

Ownership labels should be treated as protected routing data during relabel cleanup.