IT InfraTree Guides

Guides

A collection of infrastructure incident-response guides

Public guides you can read separately from solving problems. They organize the signals to check first and the recovery order by error message, symptom, checklist, and category.

Error messages

Error-message-based guides

Covers searchable errors like x509, CrashLoopBackOff, and ImagePullBackOff.

CrashLoopBackOff guide

What to check first when CrashLoopBackOff appears

For users searching for the case where a Pod restarts repeatedly and the real cause is left in the previous logs and events, this lays out the order in which to check logs and CLI output.

View guides
ImagePullBackOff guide

The registry and Secret to check first for ImagePullBackOff

For users searching for the case where the image name looks correct but the pull fails because of tag, registry credential, or network policy, this lays out the order in which to check logs and CLI output.

View guides
ImagePull cause guide

How to isolate the root cause of ImagePullBackOff

For users searching for the case where registry auth, tag drift, node egress, and mirror policy all look like the same error, this lays out the order in which to check logs and CLI output.

View guides
TLS certificate guide

How to check x509 unknown authority and missing intermediate certificates

For users searching for the case where TLS verification fails because of chain, CA bundle, or trust store differences rather than the server certificate itself, this lays out the order in which to check logs and CLI output.

View guides

Guides by symptom

Situations that look healthy but fail

Separates situations where the Pod is Running but only requests fail, or where only some clients still see the old backend after a DNS change.

Service traffic guide

When the Pod is Running but only Service traffic fails

For users searching for the case where Pod status looks healthy but the Service, Endpoint, NetworkPolicy, or Ingress path is broken, this lays out the order in which to check logs and CLI output.

View guides
DNS failure guide

When CoreDNS is Running but only DNS lookups fail

For users searching for the case where the CoreDNS Pod looks healthy but failures occur in the kube-dns path, upstream, node-local-dns, or policy, this lays out the order in which to check logs and CLI output.

View guides
Helm values guide

How to check Helm values merge and environment override drift

For users searching for the case where GitOps sync succeeds but the actual rendered manifest differs from the expected configuration, this lays out the order in which to check logs and CLI output.

View guides
ConfigMap reload guide

When the ConfigMap changed but the Pod keeps using the old values

For users searching for the case where values don't refresh after a ConfigMap update because of envFrom, subPath, volume projection, or rollout trigger differences, this lays out the order in which to check logs and CLI output.

View guides
cron failure guide

When a command works manually but fails only under cron

For users searching for the case where the same command succeeds in an interactive shell but fails only under cron or a non-interactive environment, this lays out the order in which to check logs and CLI output.

View guides
Permissions failure guide

When permissions are fixed but only new directories are created with the wrong group

For users searching for the case where existing file permissions are correct but group inheritance breaks for newly created files and directories, this lays out the order in which to check logs and CLI output.

View guides
sudo automation guide

When sudo works in the shell but fails under automation

For users searching for the case where only automation jobs fail because of TTY, sudoers, environment reset, or service user differences, this lays out the order in which to check logs and CLI output.

View guides
firewalld guide

When firewalld looks open but connections keep getting blocked

For users searching for the case where a port rule appears to exist but the connection fails because of zone, runtime/permanent drift, source binding, or an upstream firewall, this lays out the order in which to check logs and CLI output.

View guides

Checklist

A checking order operators can share right away

Organizes recurring incident-response order for deploys, cache, permissions, firewalls, and certificates.

CI/CD checklist

When GitHub Actions succeeds but only the rollout fails

For users searching for the case where the build logs look fine but the failure surfaces only on the target deploy cluster or runtime, this lays out the order in which to check logs and CLI output.

View guides
Cache failure guide

When the CI cache restores the wrong dependency graph

For users searching for the case where package.json, the lockfile, and artifacts no longer agree even after a cache hit, this lays out the order in which to check logs and CLI output.

View guides
Proxy header comparison guide

When login callbacks break due to proxy header differences between NGINX and Azure

A good fit when you first want to compare X-Forwarded-Proto, secure proxy header, callback scheme mismatch, and reverse proxy vendor differences.

View guides
OSPF check guide

A checklist for when OSPF adjacency won't form

A good fit when you first want to check OSPF adjacency down, neighbor not full, EXSTART stuck, route loss after failover, and a routing path checklist.

View guides
Routing comparison guide

How to distinguish Router-on-a-Stick from SVI in practice

A good fit when you first want to compare Router-on-a-Stick vs SVI, inter-VLAN routing design, gateway location, and the first device to check during an incident.

View guides

Category

Basic guides by area

You can skim the CI/CD, Kubernetes, Linux, Network, and Security areas first.

Linux failure guide

How to split Linux failures by systemd, permission, and filesystem signals

For users searching for the case where a service fails but you need to isolate whether the cause is a systemd unit, permission, inode, mount, or process, this lays out the order in which to check logs and CLI output.

View guides
Network failure guide

How to separate timeout and connection refused by network path

For users searching for the case where DNS, route, firewall, proxy, and listener states all look like the same connection failure, this lays out the order in which to check logs and CLI output.

View guides
Kubernetes failure guide

How to split Kubernetes failures along Pod, Service, and rollout boundaries

For users searching for the case where you need to isolate whether a Kubernetes failure started in Pod status, Service discovery, controller, storage, or network, this lays out the order in which to check logs and CLI output.

View guides
Security failure guide

How to split IAM, TLS, and WAF failures into a security operations flow

For users searching for the case where authentication, authorization, certificate, WAF, and audit log signals mix together and look like a single access failure, this lays out the order in which to check logs and CLI output.

View guides