Scenario
A label migration finishes and later one search replica remains unschedulable despite apparent cluster capacity.
An OpenSearch StatefulSet keeps one replica Pending because zone spread constraints reference the new zone label while anti-affinity still keys off the previous one
The scheduler has enough nodes, yet the replica never lands because its two placement rules now speak different zone vocabularies.
What to check first
- Identify the primary failure signal in the The Scheduler Was Told to Spread by the New Labels and Avoid by the Old Ones scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Compare all scheduler label references before adding more nodes or lowering replica counts.
Questions worth viewing together
Community-field Kubernetes problem inspired by OpenSearch and Kubernetes scheduler discussions where placement rules diverged after a zone-label migration. Unschedulable state often comes from contradictory placement logic after label migrations, not from raw capacity shortage.
Teams often blame lack of nodes when spread and anti-affinity constraints simply reference different label generations.
Label migrations should include a full scheduler-policy audit for stateful workloads.
Similar cases seen in the field