← Problem Library
Security L5 SECURITY-1395 · 11 min

A Prometheus or Alertmanager basic-auth secret rotates and one instance still...

The secret updates, yet the config reload never fires because file comparison logic ignores the semantic change under a line-ending mismatch.

SecurityPlatform ReliabilityLevel 5Pro11 min
Scenario

A basic-auth credential rotation is applied and one metrics or alerting endpoint still authenticates with the previous password.

What to check first
  • Identify the primary failure signal in the The Secret Changed and the Reloader Decided the File Looked the Same Enough scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Inspect rendered file bytes and reload triggers before recreating secrets or rolling pods.

Questions worth viewing together
What should you verify first when A Prometheus or Alertmanager basic-auth secret rotates and one instance... appears?

Community-field security problem inspired by Prometheus and Grafana rotation threads where reload logic missed an htpasswd update. Rotation bugs can hide in file rendering and comparison behavior, not only in the secret object update itself.

What usually causes A Prometheus or Alertmanager basic-auth secret rotates and one instance... in production?

Teams often blame bad secret propagation when the reloader simply never noticed the semantic file change.

What should you document after resolving A Prometheus or Alertmanager basic-auth secret rotates and one instance...?

Credential rotation tests should validate rendered-file change detection as well as secret distribution.