An API key rotation completes and later only one logging sidecar continues failing authentication despite the new secret being mounted.
An Elastic logging sidecar reconnects (elastic-sidecar-mounted-new-secret-but-kept-old-key-in-env-file)
The secret changed on disk, yet the running process still uses the boot-time materialized credential.
Scenario
What to check first
- Identify the primary failure signal in the The Secret Rotated Under the Pod and the Process Never Let Go of the Old Key scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Distinguish file refresh from process credential reload before regenerating keys again.
Questions worth viewing together
Community-field security problem inspired by Elastic and Kubernetes community threads where a rotated API key stayed stale inside a running sidecar. Projected secret updates do not guarantee the process reloads credentials it copied into env or temp files at start.
Teams often blame Kubernetes secret propagation when the process never re-read the new key from disk.
Rotation runbooks should verify application reload mechanics, not just secret object updates.
Similar cases seen in the field