Feature Article: Making Good Decisions in Highly Uncertain Situations

Introduction

At the end of December 2019, global leaders faced an environment that was becoming increasingly uncertain, due to the interaction of accelerating changes in multiple areas (e.g., technology, energy and the environment, the global economy, national security, society, and politics).

And then COVID-19 arrived, creating a global health and economic crisis, and plunging the world into what is, for many people, a situation of unprecedented uncertainty.

One of our core beliefs is that decisions should be judged not on their results (which will always be affected by both unforeseen factors and normal randomness), but on the quality of the process used to make them.

But in the fact of the complex, rapidly evolving uncertainty we face today, what constitutes a high quality decision process?

This month’s feature article provides our answer to that question.

There are many criteria that can be used to categorize different kinds of decision challenges. In our experience, the three most salient are time, the nature of goals and constraints, and the degree of uncertainty.

In situations involving high time pressure, Naturalistic Decision Making (NDM) researchers like Gary Klein and Marvin Cohen have determined that high quality decisions follow a common process:

• Using cues to recognize the situation and matching it to previous experience;
• Using that experience to quickly recall from memory a preliminary course of action (COA) that worked in the past;
• Mentally simulating whether it will achieve the minimally required goals in the current situation (i.e., satisficing);
• Implementing the COA if it does, and either modifying the initial COA or repeating the process if it does not.

For decisions in evolving situations involving multiple (and often competing) goals and complex constraints, researchers like RAND’s Robert Lempert have developed methodologies like ensemble modeling and Robust Decision Making (RDM). Other researchers continued to develop scenario planning as a decision support methodology under these same conditions.

In comparison to classic multi-attribute decision theory (which focuses on identifying the optimal COA in a relatively static situation), approaches like RDM and scenarios focus on making decisions that are “robust”. Such decisions maximize the estimated likelihood of achieving a given set of goals under a wide range of possible futures.

The nature and degree of uncertainty in a situation is another critical consideration in decision processes. Such uncertainty broadly falls into three increasingly severe categories:

Risk

• The underlying causal process generating key outcomes is understood theoretically and/or can be reliably modeled (e.g., using either classical econometric and statistical methods or via machine or deep learning);
• The distributions of possible values for the model’s parameters are well known (e.g., based on historical frequencies);
• The relationships between model outcomes and target goals is clear.

Epistemic Uncertainty

• The underlying causal process generating key outcomes is understood theoretically and can be reliably modeled;
• The distribution of parameter values is not well understood, but through more research it is possible to eventually understand it;
• The relationships between model outcomes and target goals is not clear, but through more research it is possible to eventually understand them.

Ontological Uncertainty

• The underlying process generating key outcomes is not fully understood theoretically and/or cannot be reliably modeled.
• This includes situations in which machine and deep learning are used to produce forecasts instead of theorizing about and modeling the causal processes that produce the outcomes of interest.
• It also includes data generating processes (like the political economy and financial markets) in which causation is complex (e.g., characterized by time delays and non-linearities) and constantly evolving.

Since the arrival of COVID-19, we have largely been operating in the Ontological Uncertainty zone.

At first, the health challenge in the face of disease was met by Naturalistic Decision Making methods, focused on COAs that satisficed versus a simple goal: Stop people from becoming infected and dying.

As the economic, social, and political costs of measures to stop people from dying increased, the goals and constraints facing many decision makers have become more complex, causing a shift from satisficing to a search for new courses of action that are robust across a range of possible future scenarios for the evolution of COVID-19 (e.g., the effect of social distancing and masks, and the timing and effectiveness of a vaccine). At the same time, we are gradually developing a better understanding of the underlying processes generating key health and economic outcomes in the novel situation we face. But, as recent events have painfully demonstrated, we still face considerable ignorance about the drivers of medium-term national security, social, political, and financial market outcomes.

Nine Steps to Better Decisions

So how do you make good decisions in the face of these conditions?

Step #1: Identify the goal(s) you’re trying to achieve, and the constraints on the possible COAs you could pursue.

Step #2: Regain your “Situation Awareness” (SA).

SA exists on three progressively more complete levels, which correspond to your ability to accurately answer three questions:

(1) What are the most important factors/variables/forces in the situation I face? In any uncertain situation produced by a complex adaptive system, there are always many. Focus on the ones that are most important, which will likely have the biggest impact on the way the situation evolves in the future.

(2) How are these factors related to each other? In complex adaptive systems, there are usually many relationships between key variables. The key to accurate forecasts is to focus on those relationships that are most important, which tend to be the ones that are changing the fastest, and/or have non-linear and/or time-delayed effects. These interactions are often the ones that generate emergent threats (and opportunities).

(3) How could the system evolve in the future, depending on how key uncertainties turn out, and/or the effects produced by possible actions you could take?


Step #3: More deeply examine your answers to the first two questions.

For each of them, start by listing the key facts (what you know to be true) that contributed to your answers about each important element and relationship.

Then list the key assumptions (what you believe to be true) that led to your answers. Where did those assumptions come from? Possible answers:

(1) Derived from theory. How reliable is the theory, and how accurately does it describe the current situation?

(2) Derived from history (e.g., a base rate). In an evolving system, history may not always be a good guide to the future.

(3) Derived from an analogy. How different is the analogous situation from the one you face today?

(4) Inferred from your previous personal experiences. How different were they from the situation you are facing?

(5) Social learning (e.g., what other people are saying/doing; the conventional wisdom or most popular narratives). To what extent are these people relying on the same set of common, but widely shared information inputs? To what extent do these common inputs conflict with your own information?

Step #4: More deeply explore the assumptions you have listed.

What is your 95% confidence interval around the “most likely” value you have estimated for an assumption?

What is the critical uncertainty or uncertainties that drive that confidence interval?

What could you do to reduce that uncertainty (e.g., collect key information, run an experiment, etc.), and narrow the width of your confidence interval?


Step #5: Use the facts and assumptions you have provided for Questions (1) and (2) to update your answer to question (3):

How could the system evolve in the future, depending on how key uncertainties turn out, and/or the effects produced by possible actions you could take? Remember that in complex adaptive systems, the distribution of possible outcomes usually follows a power law, and not the familiar “bell curve”. Put differently, they have a lot more “tail risk” than most people appreciate.

If possible, improve your predictive accuracy by combining your forecasts with those made by others using different methodologies and/or information.


Step #6: Identify possible courses of action (COAs), and forecast their outcomes versus your goals, based on your facts and assumptions about key situation factors and relationships.


Step #7: Test the robustness of each possible COA.

How much would each assumption have to change from its “most likely” value before you would choose a different COA?

The most robust COA is the one that will achieve your minimally acceptable outcomes versus your goals under the widest range of possible value for your assumptions.

Step #8: Always do a Pre-Mortem analysis.

As human beings, we have natural tendencies towards over-optimism and overconfidence that we need to offset.

Assume it is some point in the future and your preferred COA has failed. Ask why this happened. What indicators did you miss? What could you have done differently?

Based on your answers, either modify your COA or reconsider others.


Step #9: Implement your COA, while collecting key information and using it to constantly update your Situation Awareness.

Be especially alert to information that surprises you, which is a sign that your SA is incomplete. Write the information down, because you will soon forget it as your mind subconsciously tries to incorporate it into your existing mental model(s). Then take the time to consider what the surprising information means.

For more information on how to weigh new information to update your beliefs and Situation Awareness, see this post by our affiliate, Britten Coyne Partners.


Ultimately, the use of a good process to make decisions in situations of high uncertainty has two benefits. First, it raises the odd that you will make good decisions, and that over time this will result in cumulatively better outcomes for you than for people who use weaker decision processes.

Perhaps more important, good decision processes reduce your future exposure to self-destructive regret. Using a good process makes it far less likely you will torment yourself by endlessly revisiting past decisions and playing “woulda-coulda-shoulda” in your head.