← Problem Library
K8s L4 K8S 1414 · 12 min

A cert-manager renewal passes staging and fails production

The certificate request reaches the solver, yet the runtime class change quietly removed the credentials path needed for the real DNS update.

K8sPlatform ReliabilityLevel 4Pro12 min
Scenario

Staging renewals remain healthy and production DNS challenges start failing after a hardening rollout on solver pods.

What to check first
  • Identify the primary failure signal in the The Solver Switched Runtime Classes and Left Its DNS Credentials at the Door scenario.
  • Separate visible symptoms from the underlying technical dependency.
  • Describe the safest recovery path and the follow-up prevention work.
Checking checklist
  1. Summarize the current impact and the last known change.
  2. Collect direct evidence from logs, runtime state, and configuration before changing anything.
  3. Separate immediate recovery from permanent prevention work.
Recovery and prevention

Check runtime-class-specific volume behavior before rotating credentials or widening solver permissions.

Questions worth viewing together
What should you verify first when A cert-manager renewal passes staging and fails production appears?

Community-field Kubernetes problem inspired by cert-manager and runtime hardening discussions where the solver runtime class lost the DNS credentials mo... Hardening changes on challenge pods can remove mounted credentials paths even when the secret objects still exist.

What usually causes A cert-manager renewal passes staging and fails production in production?

Teams often blame expired DNS credentials when the new runtime class never mounted them into the solver pod.

What should you document after resolving A cert-manager renewal passes staging and fails production?

RuntimeClass changes should be validated on ephemeral challenge jobs, not only long-lived workloads.