Scenario
A managed cluster upgrade looks mostly successful and later one node pool remains disconnected from the management plane.
A Rancher-managed cluster upgrade completes and one node pool never rejoins
The control plane is healthy, yet one pool cannot reconnect because its agent manifest still references a pre-rotation secret object.
What to check first
- Identify the primary failure signal in the The Cluster Upgraded and One Pool Came Back Asking for a Secret That No Longer Existed scenario.
- Separate visible symptoms from the underlying technical dependency.
- Describe the safest recovery path and the follow-up prevention work.
Checking checklist
- Summarize the current impact and the last known change.
- Collect direct evidence from logs, runtime state, and configuration before changing anything.
- Separate immediate recovery from permanent prevention work.
Recovery and prevention
Compare mounted secret references against current management-plane token objects before recreating nodes.
Questions worth viewing together
Community-field Kubernetes problem inspired by Rancher community patterns where one node pool mounted an old bootstrap secret after rotation. Managed cluster reconnect failures often live in agent bootstrap references rather than in kubelet health.
Teams often blame bad node registration when the agent manifest still points at a deleted bootstrap secret.
Token rotations during upgrades should include validation of every agent manifest and node pool template.
Similar cases seen in the field