Symptom384 problems· 9 reviewed

Rollout Stuck

384 incident problems that show up as “Rollout Stuck”.

All problems (384)

LINUX-153A profile script exports a locale unavailable on one batch host, and only the nightly report job starts misparsing decimal inputEverything interactive seems normal, but one noninteractive environment inherits a broken locale assumption.LinuxIntermediate14 minProK8S-153A node selector matches the new GPU node pool label, but taints were added during bootstrap and the workload never lands thereThe scheduler finds the right nodes by label, yet another admission rule still excludes them.KubernetesIntermediate15 minProCICD-381A pipeline pins the runner image while the fetched bootstrap script still resolves the latest policy bundle from a mutable channel during a staged decommissionThe execution environment is frozen and its policy input is not. The service still works through the primary path, but one dependency only fails when the old component is finally drained away.CI/CDIntermediate15 minProCICD-140A release notes bot creates the Git tag before the artifact upload finalizes, and downstream consumers fetch a version with no binaries yetThe version appears published to automation, but its payload is still in flight when the deployment job tries to consume it.CI/CDIntermediate15 minProLINUX-158A systemd path unit watches for a file create, but the app replaces the file atomically and only modify events ever happenAutomation should fire, yet the real application update pattern doesn't match the chosen watcher trigger.LinuxIntermediate15 minProCICD-156A test flake suppressor matches any timeout string, and the real deadlock regression gets auto-muted during release qualificationNoise reduction automation masks the exact new symptom the release gate was meant to catch.CI/CDIntermediate15 minProNETWORK-157A Wi-Fi controller advertises the correct guest SSID, but the upstream switch trunk prunes the new VLAN after a template reapplyClients can associate, yet the data plane for the new segment never exists end to end.NetworkIntermediate15 minProNETWORK-118LLDP-MED advertises the voice VLAN, but the access switch data policy overrides it after a bouncePhones boot into the right segment once, then move unexpectedly because another edge policy reclassifies the port after link renegotiation.NetworkIntermediate15 minProCICD-149The build step caches test reports by branch name, and a recreated hotfix branch reuses stale pass results from the previous incidentNo tests actually ran for the new code, but the cache key made the workflow appear green.CI/CDIntermediate15 minProCICD-198A branch-based cache key reuses stale test evidence during a failover rehearsalThe workflow turns green even though the current code path never executed fresh validation. Normal traffic masked the issue until the standby or alternate path became active under rehearsal conditions.CI/CDIntermediate16 minProCICD-399A broken-glass deploy skips manual approval while cleanup automation still assumes the abandoned candidate release owns the shared sandbox during a staged decommissionThe emergency path works, but cleanup logic later tears down the wrong environment state. The service still works through the primary path, but one dependency only fails when the old component is finally drained away.CI/CDIntermediate16 minProCICD-131A change-approval webhook retries on timeout and opens duplicate maintenance windows for the same deploymentOne release becomes two approved windows because the idempotency key was not preserved across retries.CI/CDIntermediate16 minProK8S-150A ConfigMap reload sidecar watches inode changes, but the projected volume updates by symlink swap and the app never reloadsConfiguration changes exist in the pod, yet the watcher logic does not observe the form of filesystem mutation Kubernetes actually uses.KubernetesIntermediate16 minProK8S-147A Deployment uses maxUnavailable zero, but node pressure evicts old pods anyway and the rollout briefly drops below the intended floorThe rollout strategy is conservative, yet external eviction pressure overrides the update assumptions.KubernetesIntermediate16 minProCICD-159A release pipeline reuses the same artifact name across two environments, and the staging package overwrites the production-ready build in shared storageThe objects are all present, but one naming collision turns an environment boundary into a storage race.CI/CDIntermediate16 minProCICD-152A shared action pins the Node runtime version, but the hosted runner image removes that runtime minor line and every workflow begins failing at startupNo application code changed, yet the execution substrate stopped honoring the action's implicit runtime dependency.CI/CDIntermediate16 minProCICD-318A shared workflow pins an action SHA while the action still downloads a floating helper tool version at runtime during a failover rehearsalThe action reference looks immutable, yet part of the execution path still depends on mutable remote state. Normal traffic masked the issue until the standby or alternate path became active under rehearsal conditions.CI/CDIntermediate16 minProK8S-155An app depends on downward API labels, but the rollout controller changes the label key and the pod keeps reading an empty file without crashingThe deployment succeeds, yet the runtime configuration silently degrades because one metadata contract changed.KubernetesIntermediate16 minProK8S-270An inotify-based sidecar watches the wrong inode set after a projected ConfigMap update swaps symlinks during a failover rehearsalConfiguration changes are present while the watcher logic never sees the right file system events. Normal traffic masked the issue until the standby or alternate path became active under rehearsal conditions.KubernetesIntermediate16 minProLINUX-115GRUB saved_entry points at the wrong kernel after package install changes the menu orderThe default kernel appears pinned, but the bootloader restores a different entry because the persistent selector tracks index semantics the team forgot about.LinuxIntermediate16 minPro