"The potential for catastrophic outcome is a hallmark of complex systems"

This is primarily a set of quotes from a great (and short) manifesto-like paper "How Complex Systems Fail" by late Dr. Richard I. Cook.

I fully agree with the points gathered in that paper, so I thought I'd save it here along with most powerful (IMO) quotes.

And as another reference, this paper came to my attention via another article, "Not causal chains, but interactions and adaptations", which IMO is a little vague but still touches an important point that the Root Cause Analysis (RCA) is an obsolete and misleading practice that masks the actual nature of failures in any complex system.

So here goes.

"The complexity of these systems makes it impossible for them to run without multiple flaws being present ... The failures change constantly because of changing technology, work organization, and efforts to eradicate failures"

"The potential for catastrophic outcome is a hallmark of complex systems"

"no isolation of the ‘root cause’ of an accident is possible. The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes."

"Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved."

"all practitioner actions are actually gambles, that is, acts that take place in the face of uncertain outcomes ... in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated. "

"Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure."

And one more, not as a reason against new technology, but as an observation of potential impact that needs to be considered:

"When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures. Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology."