InsightFab
Knowledge Base/Multiple Testing Problem: FDR Control and Bonferroni Correction
Statistical Analysis6 min read

Multiple Testing Problem: FDR Control and Bonferroni Correction

This article addresses the common yet often overlooked 'multiple testing problem' in statistical analysis. The author recounts a real-world scenario where new machine consumables led to alarming CPK report figures, initially causing concern about material quality. However, the article explains that the issue often stems from the statistical nature of multiple testing, where the cumulative probability of false positives rises significantly when numerous parameters are simultaneously evaluated.

The Day the CPK Report Came Out, the Room Was Silent for Three Seconds

I remember years ago, our machines received a batch of new consumables, and everyone was very optimistic. However, when the first CPK report was released, the meeting room fell silent. The PM's face turned ashen because the report was filled with red text, indicating that the CPK for 20 of our parameters had dropped below 1.08, meaning the yield might be catastrophic. I stared at the report, thinking, "Damn, is this consumable cursed by generations past? How can it be this bad?" The PM turned to me and asked, "Is there something wrong with this batch of materials?" My heart was pounding too, but experience told me that things weren't that simple.

What Was the Problem?

Simply put, this is the "multiple testing problem" at play. Think about it: we have 20 parameters, and each parameter undergoes a hypothesis test to determine if its CPK meets the standard. Assuming we set the alpha value at 0.05 (meaning there's a 5% chance of misjudgment), then when you perform only one test, the misjudgment rate is indeed 5%.

However, when you perform 20 independent tests, each has a 5% chance of misjudgment. Can you guess what the probability is that at least one misjudgment will occur among these 20 tests? Let's calculate it: 1 – (1 – 0.05)^20 is approximately 0.64! In other words, there's as high as a 64% chance that you'll see at least one "false red flag," but in reality, there's no problem at all. It's like buying 20 lottery tickets in a row and only winning a consolation prize on one scratch-off ticket.

So the key is that when you test many items simultaneously, the individual alpha value no longer represents the overall error rate; you will see many "false positives," wasting time chasing problems that don't exist.

How to Handle It in Practice

For this multiple testing problem, there are two commonly used methods: "Bonferroni Correction" and "FDR Control."

  1. Bonferroni Correction: This method is the simplest and most straightforward. It directly divides your alpha value by the number of tests. For instance, if we have 20 parameters and the original alpha is 0.05, the corrected new alpha becomes 0.05 / 20 = 0.0025. This means we only consider a parameter to have a real problem if its p-value is less than 0.0025. The advantage of this method is that it effectively controls the "Family-Wise Error Rate" (FWER), guaranteeing that the probability of at least one misjudgment across all tests does not exceed your set alpha. However, its drawback is that it is too conservative and might cause many truly problematic parameters to be "missed" (false negatives).

  1. FDR Control (False Discovery Rate): This method is more sophisticated; it controls the "False Discovery Rate," which is the proportion of falsely rejected null hypotheses among all rejected null hypotheses. FDR first sorts all p-values from smallest to largest and then assigns a new critical value to each p-value. This method is particularly useful in environments like semiconductor fabs that require simultaneous monitoring of numerous parameters. It is less conservative than Bonferroni, allowing us to detect more potential problems without being overwhelmed by too many false positive results.

Frankly, many statistical software packages now have these functions built-in; you just need to select the correct method, and the software will perform the calculations for you.

The Most Common Pitfall

I stepped into a major pitfall back then. In our attempt "not to miss any problems," we initially didn't apply any correction and simply focused on the raw p-values of the 20 parameters. The result was that whenever a parameter's CPK slightly dropped below 1.33 (DPMO 6210), we would immediately dispatch personnel to dismantle machines, replace parts, or even stop the production line for inspection. Ultimately, we discovered that many so-called "problems" were merely statistical illusions, leading to futile efforts driven by that 5% randomness.

Once, in pursuit of a "red flag," everyone worked tirelessly for a whole week, only to find in the end that it was merely the effect of measurement error compounded by multiple testing; the actual CPK of that parameter was perfectly fine. It was then that I truly realized that using the wrong statistical analysis method is worse than wild guessing, because you end up investing real money and human resources into it.

One Thing You Can Do Today

Next time a report comes out with a lot of red flags, first ask yourself: "How many tests were performed in this report?"

Want to try it yourself?

Every tool mentioned in this article is available on InsightFab — just upload a CSV to analyze.

Go to Tools

More in this category

Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships

This article sheds light on a common pitfall in engineering data analysis: the misapplication of correlation coefficients. It explains why relying solely on Pearson correlation, which primarily detects linear relationships, can lead to incorrect conclusions when data exhibits non-linear, monotonic trends, especially when addressing issues like unexpected product yield drops.

Post-hoc Test: Choosing Between Tukey vs Bonferroni

This article addresses a common production line challenge: identifying the specific source of a problem when overall metrics like CPK fall short. It introduces Post-hoc Tests as a crucial tool to pinpoint significant differences between individual machines or groups, enabling precise problem-solving and immediate action.

Statistical Power: How to Ensure Your Conclusions Are Meaningful

This article addresses a common challenge for engineers: drawing conclusions solely based on observed data changes without accounting for statistical significance. It emphasizes the importance of using 'Statistical Power' to validate findings, ensuring that conclusions are robust and well-supported, particularly when presenting to management.