Finally I’ve read some news on what caused the North American blackout last August. According to this news article, it was a glitch in the software running on a GE Energy XA/21 server in Ohio (and, yes, it all did originate in Ohio – and wasn’t the result of some mystical lightning strike just “North of the border”). There was a failure in one of the plant’s systems but, because of the glitch, no alarms went off and the data that the technicians were seeing and thought were representative of the real-time situation were in fact out of date. Had the alarms gone off, and the system shown what was actually happening, then things would have been corrected in time before they got out of hand.

What’s kind of amusing is that when the main server went offline, a backup server came up – but it was unable to deal with the number of unprocessed events from the main server so it, then, went down as well. (I suppose what’s even more amusing is that it was the company that “brings good things to light” that inadvertantly caused the extended period of darkness. I guess that they can now legitimately claim to do both!)

Apparently, the server software’s already been patched to correct the problem. However, the Ohio plant has it on the books now to switch over to a competing monitoring solution – something that had actually been on the books prior to the blackout, but which hadn’t actually happened. (It seems as if this sort of thing – administrative delay I mean – happens in companies all over the world.)