Scenario
A zone-label migration completes and later one search replica remains Pending despite enough nodes across zones.
An OpenSearch StatefulSet keeps a replica Pending because topology spread keys off the new zone label while anti-affinity still matches the previous label generation
The scheduler has capacity, yet the pod cannot be placed because its rules now speak two different zone vocabularies.
What to check first
- Identify the primary failure signal in the The Scheduler Was Told to Spread by the New Labels and Avoid by the Old Ones scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Compare all placement rules after the label migration before scaling the cluster or reducing replicas.
Questions worth viewing together
Community-field Kubernetes problem inspired by OpenSearch and scheduler discussions where topology spread and anti-affinity referenced different zone la... Unschedulable pods after migrations often come from contradictory placement logic rather than raw capacity shortage.
Teams often blame insufficient nodes when the scheduler rules simply reference different label generations.
Label migrations should include a scheduler-policy audit for every stateful workload.
Similar cases seen in the field