Credential rotation finishes and later one ingestion path continues to return 401 despite the new secret being present in the cluster.
An Elastic API key rotation is successful and one Beats sidecar still fails because the projected secret volume never refreshed for the pod that did not roll
The cluster secret is updated, yet one workload keeps reading the old projection and stays unauthorized.
Scenario
What to check first
- Identify the primary failure signal in the IAM scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Validate rollout state and mounted secret versions before creating another key.
Questions worth viewing together
Community-field security problem inspired by Elastic community patterns where a Beats sidecar kept using an old secret projection after rotation. Secret rotation is incomplete until every consumer reloads or remounts the new material.
Teams often blame API key scope when the pod is still consuming yesterday's projection.
Projected-secret consumers need rollout verification as part of credential rotation playbooks.
Similar cases seen in the field