A credential rotation completes and one log shipping path still fails with unauthorized errors even though the cluster secret was updated everywhere.
An Elastic ingest pipeline keeps 401ing (elastic-sidecar-kept-using-projected-old-api-key-volume)
The new key exists, yet one workload keeps failing because its projected secret volume belongs to a pod template that never rolled.
Scenario
What to check first
- Identify the primary failure signal in the IAM scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Confirm pod-template rollout and secret volume refresh before minting another key.
Questions worth viewing together
Community-field security problem inspired by Elastic community patterns where a sidecar kept using an old projected API key after rotation. Cluster secret updates do not automatically mean every pod reopened the new secret material.
Teams often blame API key scope when the stale consumer never rolled onto the new projection.
Credential rotations should be tied to rollout verification, not just secret update completion.
Similar cases seen in the field