Inferential Statistics • Lesson 13
p-value & Significance Level
Learn how p-values measure evidence against the null hypothesis and how the significance level helps us make a statistical decision.
What You Will Learn
Understand what a p-value represents.
Understand the significance level α.
Learn how to compare p-value with α.
Know when to reject or fail to reject H₀.
Avoid common mistakes when interpreting p-values.
Apply p-values to real-world analytics decisions.
Why Do We Need a p-value?
In hypothesis testing, we collect sample data and calculate a test statistic. But we still need a way to describe how unusual the observed result would be if the null hypothesis were the reference model.
The p-value provides that evidence measure. A smaller p-value means the observed result is more difficult to explain under the null hypothesis.
Reference assumption
Observed evidence
Measures extremeness under H₀
STEP 1
What Is a p-value?
The p-value is the probability, calculated under the null hypothesis, of obtaining a test statistic at least as extreme as the one observed, in the direction specified by the alternative hypothesis.
In simpler words:
The p-value tells us how surprising the observed sample result would be if H₀ were the reference model.
Important
What a p-value Does NOT Mean
❌ p-value is NOT the probability that H₀ is true.
❌ p-value is NOT the probability that the observed result happened by chance.
❌ A small p-value does NOT tell us how large or important the practical effect is.
✓ It measures how compatible the observed result is with H₀, using the specified test procedure.
STEP 2
Understanding Small and Large p-values
Stronger evidence
A small p-value means the observed result would be relatively unusual if H₀ were the reference model.
p = 0.003
Weaker evidence against H₀
A larger p-value means the observed result is not especially unusual under H₀.
p = 0.42
STEP 3
What Is the Significance Level?
The significance level, written as α (alpha), is a threshold chosen before the hypothesis test.
It represents the maximum Type I error rate we are willing to tolerate under the testing framework.
Common choice
α = 0.05
Another choice
α = 0.01
Another choice
α = 0.10
STEP 4
p-value vs α
| Condition | Decision | Meaning |
|---|---|---|
| p ≤ α | Reject H₀ | Evidence is sufficiently inconsistent with H₀ under the chosen threshold. |
| p > α | Fail to reject H₀ | Evidence is not sufficiently inconsistent with H₀ under the chosen threshold. |
Easy rule to remember:
Small p-value → Reject H₀
Large p-value → Fail to reject H₀
STEP 5
Worked Example
Suppose a company wants to test whether its average delivery time is different from 30 minutes.
Null hypothesis
H₀: μ = 30
Alternative hypothesis
Hₐ: μ ≠ 30
Suppose the statistical test produces
p = 0.03
Choose α = 0.05.
0.03 < 0.05
Decision: Reject H₀
At the 5% significance level, the sample provides sufficient statistical evidence to reject the null hypothesis.
Another Example
Suppose a test produces:
p = 0.18
α = 0.05
0.18 > 0.05
Decision: Fail to reject H₀
The evidence is not sufficiently strong to reject H₀ at the 5% significance level.
Visualizing the Decision
p ≤ α
Reject H₀
p > α
Fail to reject H₀
Why Does α Matter?
The significance level is chosen before the test and controls the threshold used for the decision. Changing α can change whether a given p-value leads to rejection.
| p-value | α | Decision |
|---|---|---|
| 0.03 | 0.05 | Reject H₀ |
| 0.03 | 0.01 | Fail to reject H₀ |
Notice that the same p-value can lead to different decisions when the pre-specified significance level changes. That is why α should be selected before examining the test result.
STEP 6
Statistical Significance vs Practical Importance
A statistically significant result does not automatically mean that the effect is practically important.
With a very large sample, even a small difference can produce a small p-value. Analysts should therefore consider both statistical evidence and the size and real-world importance of the effect.
Statistical significance
Is the evidence sufficiently inconsistent with H₀ under the chosen threshold?
Practical importance
Is the size of the effect meaningful in the real-world context?
Real-World Analytics
Website Conversion Rate
An e-commerce company claims that its conversion rate is 10%. An analyst wants to determine whether the true conversion rate is different from 10%.
Null hypothesis
H₀: p = 0.10
Alternative hypothesis
Hₐ: p ≠ 0.10
Test result
p = 0.012
Choose α = 0.05.
Since 0.012 < 0.05 → Reject H₀
The test provides sufficient statistical evidence, at the 5% significance level, that the population conversion rate differs from 10%.
PRACTICE
Test Your Understanding
CHECK YOUR UNDERSTANDING
What does a p-value measure?
CHECK YOUR UNDERSTANDING
If p = 0.02 and α = 0.05, what is the decision?
CHECK YOUR UNDERSTANDING
If p = 0.18 and α = 0.05, what is the correct decision?
CHECK YOUR UNDERSTANDING
Which is a common significance level?
CHECK YOUR UNDERSTANDING
Which statement about a p-value is correct?
CHECK YOUR UNDERSTANDING
If p = 0.04 and α = 0.01, what is the correct decision?
Fill in the Blank
A smaller ______ generally indicates stronger evidence against H₀.
The significance level is represented by the Greek letter ______.
If p ≤ α, we ______ H₀.
If p > α, we ______ to reject H₀.
Analytics Challenge
Evaluate a Customer Satisfaction Test
A company claims that its average customer satisfaction score is 80. An analyst performs a hypothesis test and obtains:
p = 0.041
α = 0.05
What should the analyst decide?
0.041 < 0.05
Reject H₀
At the 5% significance level, the test provides sufficient statistical evidence against H₀.
Final Challenge
Think Like a Data Analyst
A test produces a p-value of 0.08. The analyst selected α = 0.05 before running the test.
0.08 > 0.05
Decision: Fail to reject H₀
This does not prove H₀ is true. It means the evidence was not sufficiently strong to reject H₀ at the chosen 5% significance level.
Lesson 13 Summary
A p-value measures how extreme the observed result is under H₀.
A smaller p-value generally provides stronger evidence against H₀.
The significance level α is chosen before the hypothesis test.
α = 0.05 is a commonly used significance level.
If p ≤ α, reject H₀.
If p > α, fail to reject H₀.
A p-value is not the probability that H₀ is true.
Statistical significance does not automatically imply practical importance.