InsightFab
Knowledge Base/Multiple Linear Regression: Variable Selection and Handling Multicollinearity
Statistical Analysis6 min read

Multiple Linear Regression: Variable Selection and Handling Multicollinearity

This article discusses a common challenge in process improvement where advanced data models, despite high R-squared values, fail to deliver actual yield improvements due to issues like inappropriate variable selection and multicollinearity. It highlights how overcomplicating models can lead away from the root cause, offering insights to avoid common pitfalls in data analysis.

That Day the Cpk Report Was Released, and Silence Fell for Three Seconds

I recall several years ago, a new process we had was consistently stuck with its yield, and the DPMO stubbornly stayed around 6210. Daily meetings felt like debriefings, with PMs, process engineers, and equipment engineers all looking grim. At that time, my boss told me to tackle it from the data side, to see if there were any clues. Without hesitation, I grabbed every parameter I could think of—machine settings, chemical concentrations, reaction times, even operator shifts—and ran a multivariate regression directly. The report came out, astonishingly, with an R-squared of 0.95, which looked fantastic. However, the Cpk report was still 1.08, showing no significant improvement. The entire room fell silent for three seconds, filled with an atmosphere of "Are you messing with us?"

Where Did the Problem Lie? Does Your Model Truly "Understand" the Process?

To put it plainly, back then I made a common rookie mistake: thinking that the more parameters included, the better, and that a beautiful-looking model meant I understood everything. But in reality, it's like at a social event where you only like a few people, but you insist on adding a hundred unknown girls' LINE accounts, eventually not knowing who anyone is. Statistically, this is the issue of "variable selection." Some variables you include might have no impact at all, and some might even be highly correlated with each other, which is known as "multicollinearity."

Think about it: if you throw both "machine temperature setpoint" and "actual machine temperature" into your analysis, these two will most likely be highly correlated, right? If the model treats both as independent influencing factors, it will be misleading. The result is a model that looks impressive but is useless for actual process improvement, or even guides you in the wrong direction.

How to Actually Do It? Eliminate Redundant Variables Statistically

So the key is to identify variables that are truly influential and independent of each other. Frankly, there's no silver bullet, but here are a few methods you can try:

  1. Stepwise Regression: This is like letting the computer automatically filter for you. It adds or removes variables one by one and uses statistical criteria (like AIC and BIC values) to determine the best combination. However, be aware that this is not absolute; sometimes the combination it selects may not align with your process knowledge.

  1. VIF Value (Variance Inflation Factor): This is specifically used to detect "multicollinearity." Simply put, if a variable has a high VIF value (usually above 5 or 10, caution is advised), it means it has too high a correlation with other variables. At this point, you should consider removing it or seeing if another variable can replace it.

  1. Domain Knowledge: To be honest, this is the most crucial factor. Statistical tools are merely aids; your understanding of the process is key. For instance, that day, after talking with equipment engineers and process engineers, I realized that several parameters were inherently linked, and including them only created mutual interference. In such cases, it's better to trust your experience and discard variables that are "superficially effective" but "substantially useless."

For example, if your model shows that "machine A's power" and "machine B's rotational speed" both affect yield, but you know these two are actually adjusted by the same central control system, then you should consider keeping only one of them, or finding the true source variable.

The Most Common Pitfall: Getting Overexcited by a High R-squared

When I was younger, the most common mistake I made was getting overly excited when the R-squared value exceeded 0.9, feeling like a statistics god. However, the next day, when I went to validate it on-site, I found that the improvement was far less than expected. This is because the model likely contained too many "multicollinear" variables. They make the model appear to have strong explanatory power, but in reality, the contributions of these variables are overlapping or even canceling each other out.

Another pitfall is that sometimes, to make a model "look good," you force yourself to include irrelevant variables. This results in an overly complex model that loses its "interpretability." Remember, our goal is to find "causal relationships," not just "correlation." A simple yet precise model is far more useful than a complex but distorted one.

One Thing You Can Do Today

Go back, open your regression model, and calculate the VIF value for each variable.

Want to try it yourself?

Every tool mentioned in this article is available on InsightFab — just upload a CSV to analyze.

Go to Tools

More in this category

Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships

This article sheds light on a common pitfall in engineering data analysis: the misapplication of correlation coefficients. It explains why relying solely on Pearson correlation, which primarily detects linear relationships, can lead to incorrect conclusions when data exhibits non-linear, monotonic trends, especially when addressing issues like unexpected product yield drops.

Post-hoc Test: Choosing Between Tukey vs Bonferroni

This article addresses a common production line challenge: identifying the specific source of a problem when overall metrics like CPK fall short. It introduces Post-hoc Tests as a crucial tool to pinpoint significant differences between individual machines or groups, enabling precise problem-solving and immediate action.

Statistical Power: How to Ensure Your Conclusions Are Meaningful

This article addresses a common challenge for engineers: drawing conclusions solely based on observed data changes without accounting for statistical significance. It emphasizes the importance of using 'Statistical Power' to validate findings, ensuring that conclusions are robust and well-supported, particularly when presenting to management.