The (Cyber) Risk of Human Misjudgment

Posted by Geoff Stoker on Feb 14, 2019 10:16:00 AM

blog post picture 3 

You’ve likely heard of Warren Buffet, but have you heard of Charlie Munger, the slightly more irreverent half of Berkshire Hathaway’s investment leadership team? If you know of him, have you heard any of his talks? If not, you might want to consider listening to one.

Charlie Munger has been observing, reading, and thinking for several decades about human misjudgment as it applies to the world in general and business/investment management in particular. A lot of what he has to say about how to think about problems has great applicability to the world of cybersecurity.

“Personally, I’ve gotten so that I now use a kind of two-track analysis. First, what are the factors that really govern the interests involved, rationally considered? And second, what are the subconscious influences where the brain at a subconscious level is automatically doing these things — which by and large are useful, but which often misfunction. One approach is rationality — the way you’d work out a bridge problem: by evaluating the real interests, the real probabilities and so forth. And the other is to evaluate the psychological factors that cause subconscious conclusions — many of which are wrong.”

The first track is familiar to anyone who has ever tried to solve any moderately complex problem. The second track is something many of us probably think about from time to time, but likely not as part of everyday problem analysis. Charlie has accumulated a list of 25 human tendencies — “thinking errors” — and given them names of his own liking “selected to fit Munger’s notion of what makes recall easy for Munger.” Among the names he uses are: Inconsistency-Avoidance Tendency, Twaddle Tendency, and Lollapalooza Tendency. He uses them all to explore “the natural consequences of this profusion of tendencies;” namely that “cognition is ordinarily situation-dependent so that different situations often cause different conclusions, even when the same person is thinking in the same general subject area.

Useful to Cybersecurity?

Part of what is presented in the congressional report on the 2017 Equifax breach released last month might be pretty well explained using Charlie’s list. Two major findings were that Equifax did not have complete awareness of the various systems in its infrastructure and it operated patching via an “honor system.” Of the 25 tendencies in the list, several jump right out as probably having subconsciously colluded within Equifax in the lead-up to the breach: Inconsistency-Avoidance Tendency (IAT), Reward and Punishment Superresponse Tendency (RPST), Simple, Pain-Avoiding Psychological Denial (SPAPD), and Social-Proof Tendency (SPT).

Unpacking those labels a bit: IAT — the human brain is reluctant to change current habits (this is why eliminating bad habits is so rare); RPST — “the power that incentives and disincentives have on the actions of others cannot be overstated”; the honor-based patching system at best, made it easy for Equifax to fool themselves that their patching program was solid, and at worst, made some realize it wasn’t, but were incentivized (or dis-incentivized — based on your perspective) to not rock the boat too much about it; SPAPD — fixing the patching system would be hard work; it could probably wait until tomorrow…; SPT — thinking and acting like others in the organization would make it hard to rapidly change the current systems (this is a very powerful tendency as the old elevator gag makes clear).

Let me hasten to add that I don’t expect this kind of hindsight, arm-chair, Munger-assisted psych analysis to withstand very heavy scrutiny. It’s simply meant as an example of how any organization could add a step in its decision-making process to assist in identifying subconscious blind spots that provides a real increase in the chances of avoiding them.

Here’s another take on the idea: Based on the congressional report conclusion that the breach was “entirely preventable” and the many articles that followed piling on with criticism of Equifax, many people, and maybe especially tech journalists and congressional members, appear to be significantly under appraising the ethics and competence of Equifax personnel. Possibly a bit of Influence-from-Mere-Association Tendency, Excessive Self-Regard Tendency, and/or Disliking/Hating Tendency are at play in the reporting.

I’m honestly not trying to poke people in the eye with this — it just seems to add such a helpful dimension to problem analysis that it seems useful to include it.

It reminds me of an interesting discovery 10+ years ago following a cybersecurity incident involving a trojan-worm jumping between two military systems of different classification. Many (Most? All?) cybersecurity professionals know that humans (i.e. users) are the weakest link when it comes to cybersecurity. So, the assumption was that some misbehaving user (or users) was moving a USB drive between systems. How else could the trojan-worm have gotten on the higher-level classified system? In the midst of a heated discussion, someone opened up a new package of USB drives to prove a point and . . . low-and-behold, the trojan-worm was already there! Turned out that nearly 1 in 3 devices came packaged with the infection. What was at play there? Overoptimism Tendency? Excessive Self-Regard Tendency? At least we professionals were right — we were the human weak link in assuming USB drives fresh from the manufacturer were clean. (This didn’t rule out misbehaving users.).

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman; from Caltech’s 1974 commencement address

So, what is Charlies’s recommendation to try and avoid some of our share of human misjudgment? It appears to be two things: use his list as a checklist when evaluating decisions and be on the lookout for emphasizing effects from combinations of the tendencies. He’s provided a great list, but also a really good model of how to take the general observation that natural human tendencies can have a real impact on poor judgments and create our own useful list in the way that we best understand it. Charlie’s picked 25 tendencies; Wikipedia provides over 150 cognitive biases; even selecting a few to consider each time confronted with a cybersecurity decision should bear fruit.

CYR3CON provides cyber threat intelligence through advanced machine learning and data mining of deep-/dark-web information. Cybersecurity leaders benefit from exposure to actual conversations and thoughts of bad actors conveying specific real-world threats (a perspective not normally available for consideration) as a potential hedge against misjudgments caused by tendencies like Availability-Misweighing Tendency and Reason-Respecting Tendency.