InsightFab
Knowledge Base/Statistical Significance vs. Practical Significance: p < 0.05 Does Not Equal Importance
Statistical Analysis6 min read

Statistical Significance vs. Practical Significance: p < 0.05 Does Not Equal Importance

This article offers practical insights into a common pitfall in engineering analysis. The author recounts an experience where statistically significant findings (low p-values) showed no meaningful improvement in yield during a new process introduction, highlighting the critical distinction between statistical and practical significance. The piece clarifies that statistical significance only suggests a non-random difference, not its real-world importance, guiding readers to interpret data beyond p-values to avoid costly misinterpretations.

That Day the CPK Report Came Out, the Whole Room Fell Silent for Three Seconds

I still remember several years ago, when our department had just introduced a new process. Everyone was on tenterhooks, holding weekly meetings to review data. One time, RD presented a report where several parameters had p-values less than 0.001, confidently stating: "Look, the impact of these variables on yield is statistically significant!" What was the result? One parameter's CPK increased from 1.08 to 1.10, and another's DPMO dropped from 6210 to 6195. I looked at the report, then at the production line's yield, and my face went green. In the meeting room, everyone's expressions also froze, and there was three seconds of silence. Honestly, for those few seconds, I had only one thought: "Where exactly is this statistical significance significant?"

Where's the Problem?

This is where we often get confused: "statistical significance" and "practical significance" are fundamentally two different things. Statistical significance (like p < 0.05) merely tells you that the difference you observed "is unlikely to have occurred by random chance." In other words, it's just saying "this might actually have an impact, it's not just a lucky guess."

But it never tells you how "big" or how "important" this impact is, or whether it provides "substantive" help to your product, yield, or cost. Even if a parameter has an extremely small p-value, its impact on yield might only be 0.0001%. Such a difference is essentially noise for the production line; no matter how much effort you put into adjusting it, it will be a waste of time. To put it plainly, statistical significance is like detecting a weak signal, but this signal might have no practical meaning whatsoever.

What to Do in Practice?

So, every time you get data, in addition to looking at the p-value, you need to look at two more things:

  1. Effect Size: How much does this variable "actually influence"? For example, if you adjust your parameter by 1 unit, will the yield increase by 0.5% or 5%? Will DPMO drop by 10 or 1000? You need to look at specific numbers, not just whether there "is an impact." If CPK changes from 1.08 to 1.10, the effect size is essentially negligible.
  2. Industry Standards and Customer Requirements: Does your improvement meet customer specifications? Does it surpass competitors? An "improvement" from DPMO 6210 to 6195, while statistically significant, is merely a drop in the bucket if the customer requires DPMO below 1000.

So, the key is that you must combine statistical results with your process knowledge, goals, and customer requirements.

The Most Common Trap

The biggest trap I've fallen into is pulling out all variables with p-values less than 0.05, and then spending a lot of time chasing parameters that were actually insignificant. One time, the yield of one of our batches dropped slightly, and statistical analysis revealed that a less common chemical concentration showed statistical significance. At the time, I foolishly spent a whole day chasing that supplier, demanding they provide more precise batch reports. What was the result? The actual impact of that concentration variation on yield, when calculated, was only one ten-thousandth. My section chief tore into me then, saying I was wasting time on trivial matters. Honestly, if I had just asked one more question back then: "How much impact will this difference have on yield?" I wouldn't have been so foolish.

One Thing You Can Do Today

When you get a statistical report, don't just look at the p-value. First ask: "How big is this difference? Is it important to us?"

Want to try it yourself?

Every tool mentioned in this article is available on InsightFab — just upload a CSV to analyze.

Go to Tools

More in this category

Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships

This article sheds light on a common pitfall in engineering data analysis: the misapplication of correlation coefficients. It explains why relying solely on Pearson correlation, which primarily detects linear relationships, can lead to incorrect conclusions when data exhibits non-linear, monotonic trends, especially when addressing issues like unexpected product yield drops.

Post-hoc Test: Choosing Between Tukey vs Bonferroni

This article addresses a common production line challenge: identifying the specific source of a problem when overall metrics like CPK fall short. It introduces Post-hoc Tests as a crucial tool to pinpoint significant differences between individual machines or groups, enabling precise problem-solving and immediate action.

Statistical Power: How to Ensure Your Conclusions Are Meaningful

This article addresses a common challenge for engineers: drawing conclusions solely based on observed data changes without accounting for statistical significance. It emphasizes the importance of using 'Statistical Power' to validate findings, ensuring that conclusions are robust and well-supported, particularly when presenting to management.