Just a quick post today concerning a software outage that affected thousands (millions?) of systems around the world, including some of ours. On the weekend, there was a “leap second”, which is an extra second inserted into the global time system periodically to ensure that July is still at the same point in the year 5,000 years from now. A complex set of factors caused critical software libraries and services to overwhelm the CPU when the leap second occurred.
Some of our systems crashed as a result of the leap second bug. We spent part of the weekend investigating the problem and patching the software to make sure it won’t happen again.
Wired has an excellent article on the leap second crash, if you wish to learn more.