← Problem Library
K8s L6 K8S 1395 · 15 min

An OpenSearch operator rollout looks healthy and shard allocation still...

Node labels are corrected, yet an auxiliary allocation helper keeps steering shards by stale pod-side metadata.

K8sPlatform ReliabilityLevel 6Pro15 min
Scenario

A zone normalization cleanup finishes and later shards keep bouncing despite correct node labels and healthy nodes.

What to check first
  • Identify the primary failure signal in the The Cluster Learned the New Zones and the Allocation Script Trusted Old Notes on the Pods scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Trace every source of topology truth before changing shard allocation rules or node roles.

Questions worth viewing together
What should you verify first when An OpenSearch operator rollout looks healthy and shard allocation still... appears?

Community-field Kubernetes problem inspired by OpenSearch and K8s operations patterns where an allocation helper used stale pod zone annotations. Shard churn can come from helper scripts reading pod annotations long after node labels changed.

What usually causes An OpenSearch operator rollout looks healthy and shard allocation still... in production?

Teams often blame OpenSearch allocator bugs when an external helper is feeding it stale locality data.

What should you document after resolving An OpenSearch operator rollout looks healthy and shard allocation still...?

Topology-aware helpers should prefer live node labels over copied pod annotations whenever possible.