← Problem Library
Security L5 SECURITY-1368 · 12 min

A Prometheus security alert pages the platform team because relabeling removed the owner label before Alertmanager evaluated the security route tree

The alert fires correctly, yet routing is wrong because the decisive ownership label vanished upstream in relabeling.

SecurityPlatform ReliabilityLevel 5Pro12 min
Scenario

An alerting cleanup lands and later security incidents page the default platform rotation instead of the intended responders.

What to check first
  • Identify the primary failure signal in the The Alert Survived the Pipeline and the Label That Said Who Owned It Did Not scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect label state at Alertmanager input before editing routes or receivers.

Questions worth viewing together
What should you verify first when A Prometheus security alert pages the platform team appears?

Community-field security problem inspired by Prometheus and Grafana discussions where relabeling removed the owner label before Alertmanager routing. Routing failures often begin upstream in relabel logic, not inside the final route tree.

What usually causes A Prometheus security alert pages the platform team in production?

Teams often blame Alertmanager when the required ownership label is already gone before routing.

What should you document after resolving A Prometheus security alert pages the platform team?

Ownership labels should be preserved as protected data through relabeling and aggregation.