“The Safety Systems Themselves Create New Ways For Things To Go Wrong”

At the Browser, economist Tim Harford comments on Charles Perrow’s book Normal Accidents, which suggests that our technological systems growing more complex inevitably leads to greater chaos:

Tim HarfordFor him, at the time he published the first edition of this book, Three Mile Island [the nuclear core meltdown in Pennsylvania in 1979] was the definitive one. It prefigured Chernobyl. And then he revisits the subject at the end of the 1990s. The book goes through awful accidents in complex systems and explores why they happened – the human failings that go into them, the systemic consequences, the fact you could have a very small error that propagates and propagates. It’s quite a technical book, but it’s wonderful and completely compelling.

I originally read the book because I wanted to write about a particular accident. My sister is a qualified safety engineer, and she gave me a bunch of safety engineering books. But as I read Perrow’s book, I realised that it could have been written about the financial crisis. That was really shocking to me – this realisation that these banks and their interconnections were, in many ways, the same kind of system as a nuclear reactor, or at least had very important similarities.

And is there any way of avoiding this kind of disaster in future? Does the book shed any light on that?

Tim Harford: Perrow is, in many ways, a pessimist. He says that if the system is too complicated, you will have accidents. There’s nothing you can do about it. Looking back at the history of financial crises, that’s probably appropriate. But one thing that comes out of the book is the idea that we tend to make systems more complex by adding safety systems on top of them, and that the safety systems themselves create new ways for things to go wrong. That was a key problem in the financial crisis. A lot of banks were taking bets and then insuring themselves with credit default swaps (CDS). Credit default swaps were, basically, insurance contracts that banks wrote, often with [the big insurance company] AIG. Or banks were repackaging sub-prime mortgages into vehicles that were supposed to make risky loans safe. These two innovations – the packages of sub-prime loans and the credit default swaps – were both safety systems. But they were both absolutely crucial in explaining why the system blew up. I think that’s a central and really useful idea, that these safety systems are probably not helpful – and even when they are helpful, they will have unintended consequences.”

Tags: ,