InsightFab
Knowledge Base/Post-hoc Test: Choosing Between Tukey vs Bonferroni
Statistical Analysis6 min read

Post-hoc Test: Choosing Between Tukey vs Bonferroni

This article addresses a common production line challenge: identifying the specific source of a problem when overall metrics like CPK fall short. It introduces Post-hoc Tests as a crucial tool to pinpoint significant differences between individual machines or groups, enabling precise problem-solving and immediate action.

That day the CPK report came out, the whole room was silent for three seconds, then the boss asked: "So, which machine is the problem?"

That afternoon, the CPK report from the production line was released, and the whole room fell silent for three seconds. The report showed that the CPK value for one of our critical processes was 1.08, just barely making the cut, but a long way off the target of 1.33. The boss's face darkened, and he immediately asked: "So, which machine is the problem? Is it machine A, machine B, or machine C?" At this point, if all you know is that the ANOVA P-value is less than 0.05 and you just say "there's a difference," that's completely useless. You need to tell the boss exactly which two machines have a significant difference so the production line can investigate the problem. This is where Post-hoc Tests come in handy.

Where is the problem? It's not enough to know there's a difference; you need to pinpoint "who" is different.

Simply put, ANOVA can only tell you that "among these groups, at least one group is different from the others." But it won't tell you "which group" or "which groups" are different. It's like having a bunch of kids at home, and you only know one of them failed the exam, but you don't know if it's Xiao Ming, Xiao Hua, or Xiao Pang. Post-hoc Tests are designed to compare these groups pairwise to find the combinations that truly have a difference.

So the key is, you cannot directly claim that machine A and machine B are different, or machine B and machine C are different, just because ANOVA showed a P-value less than 0.05. Doing so commits the error of "multiple comparisons." Each test carries the risk of a Type I Error (Alpha Error). The more comparisons you make, the higher the cumulative probability of making this error, easily leading you to conclude a difference where none exists.

How to do it in practice? How to choose between Tukey and Bonferroni

The two most common Post-hoc Test methods are Tukey and Bonferroni.

  1. Tukey (Tukey's HSD):

* Applicable Scenarios: When your group sizes are equal, and you want to compare "all possible combinations."

* Advantages: It simultaneously adjusts the P-values for all comparison combinations, controlling the family-wise Type I error rate. It performs well in terms of statistical power, making it less likely to miss true differences.

* Example: If you have data from 300 wafers for each of your three machines (A, B, C), and you are interested in all three combinations (A vs B, A vs C, B vs C). In this case, Tukey is the preferred choice.

  1. Bonferroni:

* Applicable Scenarios: When you "only" want to compare a specific few combinations, not all combinations. Or when your group sizes are unequal.

* Advantages: It is very conservative, directly dividing your Alpha value (usually 0.05) by the number of comparisons you make. This method strictly controls Type I error.

* Example: Following the above example, suppose historical data shows that machine C has always performed best, and you only want to compare A vs C and B vs C to see if A and B can catch up to C. In this case, Bonferroni can be used. If you have 3 groups and want to compare 2 times, then your P-value must be less than 0.05 / 2 = 0.025 to be considered significant.

In other words, Tukey is like a "comprehensive survey" tool, while Bonferroni is a "precision strike" tool.

The most common pitfall: Shouting "there's a difference" just by seeing the P-value

To be honest, I also fell into this trap in my first few years. At that time, we had a new process, and the DPMO data came out at 6210, which was significantly worse than the old process's 5800. We ran ANOVA, and the P-value was 0.038, less than 0.05. I excitedly told the boss: "There's a significant difference between the new and old processes!" The boss then asked: "So, which parameter setting caused the difference?" I was stunned on the spot because ANOVA didn't tell me that at all.

Later, I learned that I should have used a Post-hoc Test. And at that time, I was only concerned about the difference between the new and old processes, so Bonferroni would have been sufficient. If I had only made one comparison, the P-value would need to be less than 0.05 to be considered different. However, if I also wanted to compare the differences between different Batches of the new process, then I would make more than one comparison, and the P-value judgment criterion would need to be stricter.

One thing you can do today

Next time you run ANOVA, don't rush to conclusions; take one more step with a Post-hoc Test to find out who the true root cause of the problem is.

Want to try it yourself?

Every tool mentioned in this article is available on InsightFab — just upload a CSV to analyze.

Go to Tools

More in this category

Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships

This article sheds light on a common pitfall in engineering data analysis: the misapplication of correlation coefficients. It explains why relying solely on Pearson correlation, which primarily detects linear relationships, can lead to incorrect conclusions when data exhibits non-linear, monotonic trends, especially when addressing issues like unexpected product yield drops.

Statistical Power: How to Ensure Your Conclusions Are Meaningful

This article addresses a common challenge for engineers: drawing conclusions solely based on observed data changes without accounting for statistical significance. It emphasizes the importance of using 'Statistical Power' to validate findings, ensuring that conclusions are robust and well-supported, particularly when presenting to management.

Introduction to Bayesian Statistics: An Intuitive Understanding of Prior and Posterior

This article provides a highly practical approach to decision-making when faced with insufficient evidence, such as when a CPK of 1.08 causes uncertainty about whether a machine or measurement instrument is at fault. It explains how past experiences or 'gut feelings' represent statistical prior probabilities, enabling a more scientific method to identify root causes rather than relying on guesswork.