Analytical Thinking

Analytical Thinking — Sample Size and Correlation vs Causation

Beyond Facts

Informed decision making requires more than gathering and evaluating facts.

Analytical Thinking Required

It requires analytical thinking to:

Systematically break information apart
Compare different aspects of a problem
Organize information to address complex, multidimensional issues

Healthy Skepticism

As a first step, exercise healthy skepticism and consider several potential causes of what you observe.

Recognize Basic Principles

One can often avoid errors in analytical reasoning by recognizing a few basic principles.

Limitations of Small Sample Sizes

Sample Size Principle

Judgments based on a large number of observations (large number of people, events, etc.) are more likely to be accurate than judgments based on a small number of observations.

University Example

Imagine you decided to get a college degree. You have narrowed your choice to University A or University B. Several friends attend each:

Friends at University A all really like the school
Friends at University B are very dissatisfied with their school

However, on your visit:

University B: good experience, liked students and professor
University A: got the brush-off from the professor, didn't "click" with students

Which School? Many people will say you should attend University B because you personally had a positive experience ("go with your gut").

Recognize Limited Duration

While that's not necessarily wrong, it is important to fully recognize the limited duration of your visit:

How much of the school did you see?
How many different classes did you sit in on?
How many students, professors, etc. did you interact with?

Friends Have More Data

In comparison, your friends at University B will have had much more extensive opportunities to interact with many others at the school over a longer time period.

Larger Sample Better

Take the time to base your decisions on a larger number of observations when possible.

Days vs Larger Numbers

Impressions based on a few days, or a few people will usually NOT be as accurate as judgments or data based on a larger number of observations.

Correlation Doesn't Imply Causality

Correlation ≠ Causation

It is important to consider how different factors are interrelated, and whether changing (increasing or decreasing) one factor may impact others.

Don't Assume Causation

Even when two factors are clearly related, do NOT immediately assume that one factor is causing the other.

Fair Comparisons

To make firm conclusions, ensure that you are making fair comparisons.

Control Group Concept

While the real world does not lend itself to carefully designed double-blind randomized clinical trials (the "gold standard" for scientific research), it is important to apply the concept of the control group when making judgments.

Pill Outcome Question

Before scientists determine that a pill causes a certain health outcome, they:

Compare outcomes for those who have taken the pill to outcomes for those who have not
Before concluding that the pill caused the difference in outcomes, they ask: Are the two groups otherwise (mostly) the same?

XYZ Supplement Example

Imagine that a base offered XYZ supplements for free to all RegAF members:

Members who took the free supplements improved their physical fitness test scores
No change in the physical fitness test scores of others

Before You Conclude

Before you conclude that XYZ supplements cause improved physical fitness, ask some questions.

Consider Self-Selection Bias

Self-Selection Bias

Consider whether the people who took XYZ supplements differed in important ways from those who didn't:

Maybe there was a self-selection bias — members who exercised more or ate healthier foods (i.e., were more motivated to improve their fitness) were more likely to choose to take Vitamin XYZ
Maybe Vitamin XYZ takers were younger than non-takers
Or less likely than non-takers to have pre-existing health conditions

Group Similarity

The more similar the two groups, the more reasonable it would be to conclude that XYZ supplements cause improved physical fitness.

Fair Comparison

Make Fair Comparison

Deciding to take XYZ supplements may or may not be a good idea. Before you make that decision though, consider whether you're making a fair comparison.