← Problem Library
Security L6 SECURITY-1385 · 11 min

A Prometheus exporter secret is rotated and alerting still uses the old...

The secret volume updates, yet the live config never changes because reload is wired to the wrong file layer.

SecurityPlatform ReliabilityLevel 6Pro11 min
Scenario

Exporter credentials are rotated and metrics suddenly fail only for pods that rely on sidecar-generated config files.

What to check first
  • Identify the primary failure signal in the The Secret Changed and the Reloader Watched the File That Never Did scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Map the actual reload target before recreating secrets or restarting every pod.

Questions worth viewing together
What should you verify first when A Prometheus exporter secret is rotated and alerting still uses the old... appears?

Community-field security problem inspired by Prometheus and Kubernetes threads where reload watched the generated config rather than the mounted secret. Hot-reload automation can succeed technically while watching the wrong artifact layer entirely.

What usually causes A Prometheus exporter secret is rotated and alerting still uses the old... in production?

Teams often blame secret propagation when the reloader is attached to a generated file that never updates.

What should you document after resolving A Prometheus exporter secret is rotated and alerting still uses the old...?

Rotation workflows should verify which file layer the application and reloader actually consume.