Scenario
You performed a hypothesis test, and the yield difference between two production lines yielded P = 0.003, which is highly significant. Your manager asks: "How much is the difference? Do we need to stop Line B for improvement?" You examine the data: Line A's yield is 98.3%, Line B's yield is 98.1%.
The P-value is significant, but the difference is only 0.2%. Is it worth a major action?
Limitations of P-values
A P-value answers: "If there were truly no difference between the two groups, what is the probability of observing a difference this large?"
When the sample size is large enough, even a very small difference can result in a very small P-value. P < 0.05 merely indicates that the difference is not random; it does not imply practical significance.
What is Effect Size
Effect Size quantifies the practical magnitude of a difference, independent of sample size.
Common metrics:
Cohen's d (for comparing two group means)
d = (μ₁ - μ₂) / Pooled Standard Deviation
| d Value | Interpretation |
|---|---|
| 0.2 | Small effect |
| 0.5 | Medium effect |
| 0.8 | Large effect |
η² (ANOVA, proportion of variance explained)
| η² Value | Interpretation |
|---|---|
| 0.01 | Small effect (1% variance) |
| 0.06 | Medium effect (6% variance) |
| 0.14 | Large effect (14% variance) |
r (correlation coefficient, also an effect size)
| r Value | Interpretation |
|---|---|
| 0.1 | Small effect |
| 0.3 | Medium effect |
| 0.5 | Large effect |
Four Combinations of P-value + Effect Size
| P-value | Effect Size | Interpretation |
|---|---|---|
| Significant | Large | Difference is real and important; action needed |
| Significant | Small | Difference is real but unimportant; sample size may be too large |
| Not Significant | Large | Sample size may be too small; difference might truly exist |
| Not Significant | Small | No difference, or difference has no practical significance |
Application in a Factory Scenario
Scenario: Comparing Cpk of old and new processes
- P = 0.02 (Significant)
- Cohen's d = 0.15 (Small effect)
- Cpk improved from 1.40 to 1.43
→ The difference is statistically significant, but practically very small. Is the cost of process improvement worth this 0.03 Cpk increase? This is a management decision, not a statistical one.
Golden Quote
"P-values tell you if a difference is real; effect sizes tell you if a difference is important—making decisions based solely on P-values is letting statistics make your business judgment for you."