Inferential Statistics • Lesson 19
Type I & Type II Errors + Statistical Power
Learn what can go wrong when making statistical decisions, how Type I and Type II errors differ, and how statistical power helps us detect real effects.
What You Will Learn
1. Statistical Decisions Are Not Perfect
Hypothesis testing uses sample data to make a decision about a population. Because samples contain randomness, a statistical decision can sometimes be wrong.
There are two fundamental types of errors:
Type I Error
Rejecting H₀ when H₀ is actually true.
Type II Error
Failing to reject H₀ when H₀ is actually false.
2. The Decision Matrix
Think of hypothesis testing as a decision made under uncertainty.
| Reality | Fail to Reject H₀ | Reject H₀ |
|---|---|---|
| H₀ is true | Correct decision | Type I Error |
| H₀ is false | Type II Error | Correct decision |
3. Type I Error — False Positive
A Type I error occurs when we reject the null hypothesis even though the null hypothesis is actually true.
Type I Error = Reject a true H₀
It is often described as a false positive: the analysis indicates evidence of an effect even though the assumed null condition is actually true.
Example: Website Conversion
Suppose H₀ states that a new website design has the same conversion rate as the old design.
If the analysis rejects H₀ and concludes that conversion has changed, but in reality there is no population-level change, that decision is a Type I error.
4. Significance Level α
The significance level, written as α, is the chosen threshold for controlling the Type I error rate under the testing procedure.
α = 0.10
10% threshold
α = 0.05
5% threshold
α = 0.01
1% threshold
Important:
α should generally be chosen before examining the test result. A smaller α makes the rejection threshold more stringent and, all else equal, reduces the Type I error rate.
5. Type II Error — False Negative
A Type II error occurs when we fail to reject H₀ even though H₀ is actually false.
Type II Error = Fail to Reject a false H₀
It is often described as a false negative: a real effect exists, but the statistical procedure does not detect enough evidence to reject H₀.
Example: Marketing Campaign
Suppose a new marketing campaign really does increase the population conversion rate.
If a study fails to reject H₀ and concludes that there is insufficient evidence of an increase, that could be a Type II error if the true population effect exists.
6. β — Probability of Type II Error
β = P(Type II Error)
β describes the probability of failing to reject H₀ for a particular alternative situation where H₀ is false.
Unlike α, β is not usually a single fixed number for an entire hypothesis test. It depends on factors such as the true effect size, sample size, variability, and the significance level.
7. Statistical Power
Statistical power is the probability that a test correctly rejects H₀ when a specified alternative condition is true.
Power = 1 − β
Higher power means a greater probability of detecting a real effect of the specified size under the assumed conditions.
Low Power
Greater chance of missing a real effect
Higher Power
Better ability to detect a specified real effect
Power = 1 − β
Power and Type II error move in opposite directions
Power Example
Suppose a study has β = 0.20 for a particular true effect size.
Type II error probability
β = 0.20
Statistical power
1 − 0.20 = 0.80
So the power is 80% for that specified alternative condition.
8. What Affects Statistical Power?
Sample Size
Larger samples generally provide more information and can increase power.
Effect Size
Larger true effects are generally easier to detect than smaller effects.
Variability
Lower variability generally makes a given effect easier to detect.
Significance Level
Increasing α generally increases power, but also increases the Type I error rate.
9. Why Sample Size Matters
A larger sample often reduces sampling uncertainty. This can make a genuine effect easier to distinguish from random variation.
n = 25
More sampling uncertainty
n = 100
Less sampling uncertainty
n = 400
Still less sampling uncertainty
Increasing sample size does not automatically guarantee high power. Power still depends on the effect size, variability, α, test design, and other assumptions.
10. Type I vs Type II Error
| Feature | Type I Error | Type II Error |
|---|---|---|
| Decision | Reject H₀ | Fail to reject H₀ |
| Reality | H₀ is true | H₀ is false |
| Common description | False positive | False negative |
| Symbol | α | β |
| Related concept | Significance level | Power = 1 − β |
Real-World Analytics Example
A/B Testing
An e-commerce company tests a new checkout design.
H₀:
The new checkout design does not change the population conversion rate.
Type I Error:
Concluding that the checkout design changes conversion when it actually does not.
Type II Error:
Failing to detect a real change in the population conversion rate.
Power:
The probability of detecting the specified real conversion-rate effect under the assumed conditions.
11. Choosing the Right Balance
Statistical study design involves balancing the consequences of different errors. The appropriate balance depends on the real-world cost of false positives and false negatives.
False Positive Cost
What happens if we act as though an effect exists when it does not?
False Negative Cost
What happens if we miss a real effect?
Interactive Practice
Test your understanding.
CHECK YOUR UNDERSTANDING
What is a Type I error?
CHECK YOUR UNDERSTANDING
What is a Type II error?
CHECK YOUR UNDERSTANDING
What does α represent in hypothesis testing?
CHECK YOUR UNDERSTANDING
If β = 0.20 for a specified alternative condition, what is the power?
CHECK YOUR UNDERSTANDING
Which change generally increases statistical power, all else equal?
CHECK YOUR UNDERSTANDING
If a test has high power for a specified effect, what does that mean?
Fill in the Blanks
A Type I error means rejecting a ______ null hypothesis.
A Type II error means failing to reject a ______ null hypothesis.
Statistical power equals 1 − ______.
The significance level is commonly represented by the Greek letter ______.
Data Analytics Challenge
Website Experiment
An analytics team tests whether a redesigned landing page changes the conversion rate.
H₀:
The redesign does not change the population conversion rate.
Scenario A:
The redesign actually has no effect, but the test rejects H₀.
This is a Type I error.
Scenario B:
The redesign actually improves conversion, but the test fails to reject H₀.
This can be a Type II error.
Final Challenge
Power Calculation
A study has an estimated Type II error probability of β = 0.15 for a specified alternative effect.
Calculate statistical power.
Power = 1 − β
Power = 1 − 0.15 = 0.85
Therefore, power is 85% for that specified alternative condition.
Think further:
If the study needs greater power, increasing sample size is often one practical design strategy, assuming other important factors remain appropriate.
Lesson Summary
✓ Type I error means rejecting a true H₀.
✓ Type II error means failing to reject a false H₀.
✓ α is the chosen significance level.
✓ β represents the Type II error probability for a specified alternative condition.
✓ Statistical power = 1 − β.
✓ Larger samples generally increase power.
✓ Larger effects are generally easier to detect.
✓ Lower variability generally increases the ability to detect a given effect.
✓ Increasing α generally increases power but also increases the Type I error rate.
✓ Power is always interpreted relative to a specified alternative effect and study design.