Scenario
A team rotates monitoring credentials and later finds dashboards healthy while rule evaluations continue to fail with auth errors.
A Grafana datasource secret rotates successfully and alert rules still fail because the unified alerting scheduler kept the previous token in its long-lived datasource cache
Dashboards recover after rotation, yet alert executions continue failing because the scheduler process still reuses an older credential view.
What to check first
- Identify the primary failure signal in the Dashboards Saw the New Secret Before the Alert Engine Did scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Check which Grafana subsystem cached the datasource token before rotating the secret again.
Questions worth viewing together
Community-field security problem inspired by Grafana community threads where datasource token rotation broke alerts but not dashboards. Read-path recovery in the UI does not prove every background scheduler reloaded the same secret.
Teams often blame the datasource provider when only Grafana's alerting side kept the stale token.
Credential rotation should include validation across dashboards, alerting, and background evaluation workers.
Similar cases seen in the field