Linux Performance and Observability
30 incident problems about Linux Performance and Observability. Start with the reviewed ones.
Read first
How to split Linux failures by systemd, permission, and filesystem signalsAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when a service fails but you need to isolate whether the cause is a systemd unit, permission, inode, mount, or process.Linux3 min readWhen sudo works in the shell but fails under automationAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when only automation jobs fail because of TTY, sudoers, environment reset, or service user differences.Linux3 min readWhen permissions are fixed but only new directories are created with the wrong groupAn InfraTree guide that lays out the first signals to check, the CLI verification order, common misdiagnoses, and a safe recovery path when existing file permissions are correct but group inheritance breaks for newly created files and directories.Linux3 min read
Recommended problems
Reviewed problems first, then problems with detailed scenarios.
Load average is 40 but the CPU is mostly idle: I/O waitLoad average is 40 but the CPU is mostly idle: I/O wait is a hands-on troubleshooting drill. Separate CPU saturation from I/O wait when load average spikes. linux-performance-and-observability needs to be checked by narrowing scope, recent change, and the current live signal b...ReviewedLinuxIntermediate16 minFreeThe OOM killer terminates the database during a nightly batchThe OOM killer terminates the database (Resource Exhaustion) is a hands-on troubleshooting drill. Read kernel OOM logs to find which process caused memory exhaustion and why the database was chosen. linux-performance-and-observability needs to be checked by narrowing scope, re...ReviewedLinuxIntermediate18 minFree
All problems (30)
Load average is 40 but the CPU is mostly idle: I/O waitLoad average is 40 but the CPU is mostly idle: I/O wait is a hands-on troubleshooting drill. Separate CPU saturation from I/O wait when load average spikes. linux-performance-and-observability needs to be checked by narrowing scope, recent change, and the current live signal b...ReviewedLinuxIntermediate16 minFreeThe OOM killer terminates the database during a nightly batchThe OOM killer terminates the database (Resource Exhaustion) is a hands-on troubleshooting drill. Read kernel OOM logs to find which process caused memory exhaustion and why the database was chosen. linux-performance-and-observability needs to be checked by narrowing scope, re...ReviewedLinuxIntermediate18 minFreeLINUX-120A kernel livepatch package loads successfully but the target symbol version mismatch means the host remains vulnerableThe livepatch service reports success, yet the running kernel never actually applies the fix because the binary compatibility window was missed.LinuxAdvanced17 minProLINUX-127Journal forward-secure sealing is enabled, but the verification key was rotated out of sync and log integrity checks now failLogs are still written, yet the trust model behind their integrity no longer validates after a partial key update.LinuxAdvanced17 minProLINUX-129The auditd backlog limit is too small and a privilege escalation burst drops the very exec events the investigation needsThe host stays online, but the audit trail is incomplete because event pressure outran the queue design.LinuxAdvanced17 minProLINUX-393An application writes Unicode filenames correctly while a backup or sync service still runs under the old locale and drops part of the path set during a staged decommissionThe system stores the files and one scheduled path can no longer see them all. The service still works through the primary path, but one dependency only fails when the old component is finally drained away.LinuxIntermediate15 minProLINUX-146An rsyslog ruleset drops duplicate messages aggressively, and a brute-force pattern disappearsThe host is quieter, but the deduplication logic removed meaningful repetition that analysts rely on.LinuxIntermediate15 minProLINUX-102Journal rate limiting suppresses the real crash pattern and the team chases the wrong processOperators only see a handful of logs because the journal quota is dropping the repetitive fault signal during the outage window.LinuxIntermediate15 minProLINUX-330A login shell exports one locale while the noninteractive service unit inherits another and decimal parsing diverges across the same host during a failover rehearsalNothing is wrong globally; the environment contract is simply different by execution mode. Normal traffic masked the issue until the standby or alternate path became active under rehearsal conditions.LinuxIntermediate16 minProLINUX-192Noise suppression removes the repeated signal analysts rely on for brute-force detection during a failover rehearsalThe logs are cleaner and the incident pattern disappears with the noise. Normal traffic masked the issue until the standby or alternate path became active under rehearsal conditions.LinuxIntermediate16 minProLINUX-511An application writes Unicode filenames correctlyAn application writes Unicode filenames correctly focuses on linux-performance-and-observability and asks the reader to isolate Rollout Stuck. 실무에서는 linux-performance-and-observability 문제를 볼 때 서비스 로그만 보지 말고 inode, 파일시스템 여유, 포트 점유, systemd 상태, 최근 패키지 변경까지 같이 확인해야 원인을 빨리 좁힐 수 있습니다.LinuxIntermediate17 minProLINUX-118cgroup v2 memory.high throttles the workers and the service looks slow instead of obviously brokenLatency climbs without hard OOM events because the workload is being pressured by the soft limit long before memory.max is reached.LinuxAdvanced17 minProLINUX-571An application writes Unicode filenames correctlyAn application writes Unicode filenames correctly focuses on linux-performance-and-observability and asks the reader to isolate Rollout Stuck. 실무에서는 linux-performance-and-observability 문제를 볼 때 서비스 로그만 보지 말고 inode, 파일시스템 여유, 포트 점유, systemd 상태, 최근 패키지 변경까지 같이 확인해야 원인을 빨리 좁힐 수 있습니다.LinuxIntermediate18 minProLINUX-111rsyslog disk-assisted forwarding queue fills the spool partition and local logging blocks behind remote delivery pressureRemote forwarding is supposed to be resilient, yet local observability degrades because the spillover queue shares space with critical system paths.LinuxAdvanced18 minProLINUX-691An application writes Unicode filenames correctlyAn application writes Unicode filenames correctly focuses on linux-performance-and-observability and asks the reader to isolate Rollout Stuck. 실무에서는 linux-performance-and-observability 문제를 볼 때 서비스 로그만 보지 말고 inode, 파일시스템 여유, 포트 점유, systemd 상태, 최근 패키지 변경까지 같이 확인해야 원인을 빨리 좁힐 수 있습니다.LinuxIntermediate20 minProLINUX-810An application writes Unicode filenames correctlyAn application writes Unicode filenames correctly focuses on linux-performance-and-observability and asks the reader to isolate Rollout Stuck. 실무에서는 linux-performance-and-observability 문제를 볼 때 서비스 로그만 보지 말고 inode, 파일시스템 여유, 포트 점유, systemd 상태, 최근 패키지 변경까지 같이 확인해야 원인을 빨리 좁힐 수 있습니다.LinuxIntermediate23 minProLINUX-1170An application writes Unicode filenames correctlyAn application writes Unicode filenames correctly focuses on linux-performance-and-observability and asks the reader to isolate Rollout Stuck. 실무에서는 linux-performance-and-observability 문제를 볼 때 서비스 로그만 보지 말고 inode, 파일시스템 여유, 포트 점유, systemd 상태, 최근 패키지 변경까지 같이 확인해야 원인을 빨리 좁힐 수 있습니다.LinuxIntermediate29 minProLINUX-153A profile script exports a locale unavailable on one batch host, and only the nightly report job starts misparsing decimal inputEverything interactive seems normal, but one noninteractive environment inherits a broken locale assumption.LinuxIntermediate14 minProLINUX-142A log shipping agent rotates its own state file to the root partition and eventually fills the boot disk instead of the intended data mountObservability remains intact, but the persistence path for agent state quietly lives on a much smaller filesystem.LinuxIntermediate15 minProLINUX-134Coredumps are enabled, but the external coredump partition is full so every crash report silently disappearsDevelopers expect postmortem artifacts, yet the configured storage target has no remaining capacity.LinuxIntermediate15 minPro