← Problem Library
CI/CD L3 CICD-699 · 21 min

A broken-glass deploy skips manual approval

The emergency path works, but cleanup logic later tears down the wrong environment state. The service still works through the primary path, but one dependency only fails when the old component is finally drained away. The primary component still serves traffic, yet the retirement exercise exposes a latent dependency on what is being removed. The main path still works until the exercise finally removes the legacy dependency it had quietly relied on. Normal traffic stayed clean until the exercise finally removed a legacy path that the service still relied on indirectly. Normal traffic remained healthy until the exercise finally removed a legacy dependency the service still depended on indirectly. Normal traffic stayed clean until the exercise finally removed a legacy dependency the service still relied on indirectly.

CI/CDFoundation OpsLevel 3Free21 min
Scenario

The emergency path works, but cleanup logic later tears down the wrong environment state. The service still works through the primary path, but one dependency only fails when the old component is finally drained away. The primary component still serves traffic, yet the retirement exercise exposes a latent dependency on what is being removed. The main path still works until the exercise finally removes the legacy dependency it had quietly relied on. Normal traffic stayed clean until the exercise finally removed a legacy path that the service still relied on indirectly. Normal traffic remained healthy until the exercise finally removed a legacy dependency the service still depended on indirectly. Normal traffic stayed clean until the exercise finally removed a legacy dependency the service still relied on indirectly.

What to check first
  • Identify the primary failure signal in the Emergency Deploy Succeeded, Cleanup Ownership Was Wrong scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Choose the smallest safe recovery action first, then record the prevention work that reduces repeat incidents.

Questions worth viewing together
Should I translate commands, logs, or product names?

No. Keep commands, logs, file names, APIs, and product names unchanged, then explain the reasoning in the selected UI language.

What should the answer focus on?

State the root cause, the evidence that supports it, and the safest recovery direction.

Why can the scenario remain in the original language?

The source scenario is treated as an incident artifact. Guidance, checklist, hints, and explanations can be localized around it.