Scenario
An alerting pipeline cleanup merges relabel rules and later some incidents stop paging the owning team even though alerts still fire.
A Prometheus alert reaches Alertmanager and never pages the right team because the alert relabel stage removed the `owner` label before routing rules executed
The alert is real, yet it falls into the default receiver because the routing label vanished in post-processing.
What to check first
- Identify the primary failure signal in the The Alert Fired and the Label the Pager Needed Was Stripped on the Way Out scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Compare labels before and after relabeling before rewriting Alertmanager routes.
Questions worth viewing together
Community-field security problem inspired by Prometheus and Grafana alerting cases where relabeling removed the owner label before routing. Routing failures often come from relabel stages, not from the alert expression or receiver integration.
Teams often blame Alertmanager route order when the label it was routing on never made it that far.
Relabel changes should be validated against downstream routing rules and escalation ownership, not only alert firing.
Similar cases seen in the field