Rollback and Rollout
22 incident problems about Rollback and Rollout. Start with the reviewed ones.
Read first
When the CI cache restores the wrong dependency graphAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when package.json, the lockfile, and artifacts no longer agree even after a cache hit.CI/CD3 min readWhen GitHub Actions succeeds but only the rollout failsAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when the build logs look fine but the failure surfaces only on the target deploy cluster or runtime.CI/CD3 min readThe checking order when GitHub Actions succeeds but only the deploy failsAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when the workflow is green but the failure happens only in the rollout, promotion, or health check stage.CI/CD3 min read
Recommended problems
Reviewed problems first, then problems with detailed scenarios.
CICD-001Finding the real failure point in GitHub Actions build logsA training scenario for isolating the true cause from logs where npm dependency conflicts and cache traces are mixed together.ReviewedCI/CDIntermediate20 minFreeCICD-002Recovering a container deployment where the latest tag blocks rollbackCovers how to redesign a safe rollback structure in a pipeline that has no image versioning strategy.ReviewedCI/CDAdvanced25 minProCICD-009Deployment failingA scenario that narrows down the root cause, centered on checking the permission-policy difference by event type, in the situation of a deployment failing because the secret is empty only on tag releases.ReviewedCI/CDIntermediate21 minFreeCICD-012Tests pass but the migration order is wrong, causing an incident right after deployTests pass but the migration order is wrong, causing an incident right is a hands-on troubleshooting drill. Covers an incident caused by the application deploy and database schema change happening in the wrong order. GitHub GitHub Actions Workflows needs to be checked by narro...CI/CDIntermediate24 minProCICD-016Missing feature-flag verification exposes inactive code directly in productionMissing feature-flag verification exposes inactive code directly in production is a hands-on troubleshooting drill. Tracks a feature-flag operations gap that was in the release plan but missing from the actual verification flow. GitHub Actions Workflows needs to be checked by...CI/CDAdvanced23 minProCICD-019In a blue-green deploy, traffic is switched but the background worker stays on the old versionA situation where success was judged from the web-traffic switch alone, but the async worker remained on the previous version.CI/CDAdvanced28 minPro
All problems (22)
CICD-009Deployment failingA scenario that narrows down the root cause, centered on checking the permission-policy difference by event type, in the situation of a deployment failing because the secret is empty only on tag releases.ReviewedCI/CDIntermediate21 minFreeCICD-001Finding the real failure point in GitHub Actions build logsA training scenario for isolating the true cause from logs where npm dependency conflicts and cache traces are mixed together.ReviewedCI/CDIntermediate20 minFreeCICD-056CloudFront invalidation runs before the new asset manifest is fully publishedThe deployment pipeline clears the CDN correctly, but users still receive broken bundles because the invalidation precedes the final asset upload and manifest sync.CI/CDIntermediate18 minFreeCICD-030Release note generation fails on shallow clone runnersRelease note generation fails on shallow clone runners is a hands-on troubleshooting drill. A changelog step depends on full git history, but shallow checkout settings hide the required tags and commits. GitHub GitHub Actions Workflows needs to be checked by narrowing scope, r...CI/CDIntermediate18 minFreeCICD-002Recovering a container deployment where the latest tag blocks rollbackCovers how to redesign a safe rollback structure in a pipeline that has no image versioning strategy.ReviewedCI/CDAdvanced25 minProCICD-016Missing feature-flag verification exposes inactive code directly in productionMissing feature-flag verification exposes inactive code directly in production is a hands-on troubleshooting drill. Tracks a feature-flag operations gap that was in the release plan but missing from the actual verification flow. GitHub Actions Workflows needs to be checked by...CI/CDAdvanced23 minProCICD-012Tests pass but the migration order is wrong, causing an incident right after deployTests pass but the migration order is wrong, causing an incident right is a hands-on troubleshooting drill. Covers an incident caused by the application deploy and database schema change happening in the wrong order. GitHub GitHub Actions Workflows needs to be checked by narro...CI/CDIntermediate24 minProCICD-019In a blue-green deploy, traffic is switched but the background worker stays on the old versionA situation where success was judged from the web-traffic switch alone, but the async worker remained on the previous version.CI/CDAdvanced28 minProCICD-008Deployments passing without verificationA scenario that narrows down the root cause, centered on designing governance that enforces the verification step before deployment approval, in the situation of deployments passing without verification because a smoke-test conditional is wrong.CI/CDAdvanced23 minProCICD-058Production environment review waits foreverThe environment requires human approval on paper, but tagged releases stall indefinitely because the reviewer and branch rules were designed only for branch-based promotion.CI/CDAdvanced24 minProCICD-029Monorepo path filter misses shared library changesMonorepo path filter misses shared library changes is a hands-on troubleshooting drill. Only some services rebuild because the path filter ignores a shared package that affects multiple deployments. GitHub Rollback and Rollout needs to be checked by narrowing scope, recent cha...CI/CDAdvanced26 minProCICD-036CodeDeploy rollback alarm never firesThe blue-green deployment partially shifts traffic, but the rollback condition never triggers because the health alarm watches only the stable target group.CI/CDAdvanced27 minProCICD-052Reusable workflow caller changes the OIDC subject and breaks deploy role trustThe same deployment worked as a repository-local workflow, but it fails after moving into a reusable workflow because the token subject no longer matches the original trust condition.CI/CDAdvanced27 minProCICD-032OIDC subject condition mismatch denies AWS deploy role assumptionGitHub Actions reaches AWS STS, but the trust policy rejects the web identity token because the subject or audience condition no longer matches the branch and environment path.CI/CDAdvanced28 minProCICD-026Promotion pipeline deploys stale image from previous commitThe promotion stage uses an artifact reference that looks correct but resolves to an older image digest after registry cleanup.CI/CDAdvanced29 minProCICD-040Terraform apply succeeds in staging but production backend locks a different state tableThe same pipeline logic passes against one workspace, but production never converges because the remote backend, state lock table, or workspace mapping differs subtly.CI/CDAdvanced30 minProCICD-011Branch protection rules exist, but the deploy workflow pushes directly to mainCovers a deployment-policy problem where code-review protection exists but the automation account becomes a bypass path.CI/CDBeginner17 minProCICD-039ECR lifecycle cleanup deletes digest still referenced by promotion manifestThe production promotion manifest points to an immutable digest, but registry cleanup removes the only copy before the delayed deployment window starts.CI/CDIntermediate22 minProCICD-037CloudFormation change set succeeds but rollback fails on nested stack driftThe proposed change set looks safe, yet rollback breaks because one nested stack already drifted away from the expected template state.CI/CDAdvanced29 minProCICD-060Terraform PR plan is green but main apply uses a different variable setReviewers trust the plan output, but production apply behaves differently because the PR stage reads one workspace and tfvars path while the main branch apply uses another.CI/CDAdvanced29 minPro