Systems
The Watchdog That Watched a Disk Die Anyway
A disk-space guard observed the exact failure condition it was built to catch, twice, and the disk hit zero nine minutes later. Two bugs, two fixes, three hours apart.
One Flake, Seven Hosts, and the Backup That Died Without Telling Anyone
Two hosts each carried their own copy of the same backup config, and the same structural mistake in both of them let one backup silently stop running for two days.