Scenario
One Loki distributor keeps rejecting chunks after a scale-out and gossip key rotation.
A Grafana Loki write path scales out and one distributor still rejects chunks because the ingester ring member list is cached through a stale memberlist secret
New replicas register, but one distributor still trusts the previous gossip key and never accepts the new ring view.
What to check first
- Identify the primary failure signal in the The New Writers Joined the Ring and One Distributor Kept Listening for the Password Everyone Else Had Already Changed scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Compare memberlist secrets and distributor ring caches before restarting the cluster.
Questions worth viewing together
No. Keep commands, logs, file names, APIs, and product names unchanged, then explain the reasoning in the selected UI language.
State the root cause, the evidence that supports it, and the safest recovery direction.
The source scenario is treated as an incident artifact. Guidance, checklist, hints, and explanations can be localized around it.
Similar cases seen in the field