InsightFab
Knowledge Base/Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships
Statistical Analysis6 min read

Correlation Coefficient: Pearson vs Spearman – Linear vs Monotonic Relationships

This article sheds light on a common pitfall in engineering data analysis: the misapplication of correlation coefficients. It explains why relying solely on Pearson correlation, which primarily detects linear relationships, can lead to incorrect conclusions when data exhibits non-linear, monotonic trends, especially when addressing issues like unexpected product yield drops.

The day the CpK report came out, the whole room fell silent for three seconds

Remember the last incident? The one where the wafer scrap rate on a particular machine suddenly soared from 0.5% to 2%, alarming everyone. The PM, looking grim, rushed over to ask me: "Lao Zhang, last time you said parameter A was 'highly correlated' with yield and confidently guaranteed it. Yet now the yield is this poor, and the A value is clearly still within spec. Did you make a mistake somewhere?" To be honest, my heart sank a bit at that moment. I had clearly run the Pearson correlation coefficient, and it was as high as 0.85 – such a beautiful number. How could this happen? Seeing the PM's almost fuming face, I could only try to play it off, saying: "Hmm... we might need to dig deeper into the data to see if there are other hidden variables..."

What was the problem?

Frankly speaking, you and I have probably both encountered this problem. When conducting data analysis on the factory floor, we often habitually calculate a Pearson correlation coefficient, and if the number is high enough, we feel like we've found the holy grail. But to put it plainly, the Pearson correlation coefficient only looks at "linear relationships." What does that mean? It means when one variable increases, the other variable also increases or decreases "proportionally." Imagine if you ran an extra kilometer every day, and your weight decreased proportionally. That's a linear relationship.

But why is the real world never that simple? More often, our data relationships are "monotonic" but "non-linear." What does monotonic mean? It means when one variable increases, the other variable also increases (or decreases), but the magnitude of the increase or decrease is not necessarily the same. For example, if you sleep an extra hour every day, initially, you might feel much more refreshed. But after a certain point, sleeping another hour won't significantly improve your mental state as much. This is a monotonic relationship, but it's not linear. So, the key point is, when your data relationship is not a straight line, the Pearson correlation coefficient will give you a "false impression," making you think there's no correlation or a very weak one.

How to actually do it?

At this point, we need to introduce another old friend, the Spearman correlation coefficient. It specializes in handling this type of "monotonic relationship." How does it work? To put it simply, it doesn't look at the raw values. Instead, it converts your data into "ranks" and then calculates the correlation.

Let's take a practical example:

  1. Assume you have 5 wafers today, with "exposure times" of 100, 105, 110, 120, and 130 seconds, respectively.
  2. And their corresponding "Cpk" values are 1.08, 1.25, 1.30, 1.31, and 1.15, respectively.

If you use Pearson to calculate, you might find that Cpk actually decreases when the exposure time reaches 130 seconds, leading to a correlation that is not as expected. But if you use Spearman:

  1. We rank the exposure times: 100 (1st), 105 (2nd), 110 (3rd), 120 (4th), 130 (5th).
  2. Then we rank the Cpk values: 1.08 (1st), 1.15 (2nd), 1.25 (3rd), 1.30 (4th), 1.31 (5th).

Then, calculate the correlation coefficient based on these ranks. You'll find that even if the original values don't have a strict linear relationship, as long as the trend is moving in the same direction, Spearman can detect it. So, next time you're looking at data, if you suspect a non-linear relationship, try running both!

The Most Common Pitfall

The biggest pitfall I've ever fallen into was treating the Pearson correlation coefficient as the sole criterion. That time, trying to track down a problem where DPMO surged from 6210 to 8500, I spent three days and nights. I found that a certain etching time parameter had no obvious relationship with DPMO, with Pearson showing only 0.2. I even reported to my boss, saying, "This parameter is not the main cause." But my boss, not convinced, told me to draw a scatter plot. Once I drew it, I discovered that DPMO was very low within a certain range of etching time, but beyond that range, DPMO would sharply increase, exhibiting an "L"-shaped relationship. Of course, Pearson couldn't capture this type of plot, but as soon as I ran Spearman, the correlation coefficient immediately jumped to 0.75! At that moment, I could only sheepishly admit my error and re-adjust my analytical approach. Therefore, just looking at numbers without drawing plots will definitely lead to significant losses.

One Thing You Can Do Today

Next time you run a correlation, run both coefficients and then draw a scatter plot.

Want to try it yourself?

Every tool mentioned in this article is available on InsightFab — just upload a CSV to analyze.

Go to Tools

More in this category

Post-hoc Test: Choosing Between Tukey vs Bonferroni

This article addresses a common production line challenge: identifying the specific source of a problem when overall metrics like CPK fall short. It introduces Post-hoc Tests as a crucial tool to pinpoint significant differences between individual machines or groups, enabling precise problem-solving and immediate action.

Statistical Power: How to Ensure Your Conclusions Are Meaningful

This article addresses a common challenge for engineers: drawing conclusions solely based on observed data changes without accounting for statistical significance. It emphasizes the importance of using 'Statistical Power' to validate findings, ensuring that conclusions are robust and well-supported, particularly when presenting to management.

Introduction to Bayesian Statistics: An Intuitive Understanding of Prior and Posterior

This article provides a highly practical approach to decision-making when faced with insufficient evidence, such as when a CPK of 1.08 causes uncertainty about whether a machine or measurement instrument is at fault. It explains how past experiences or 'gut feelings' represent statistical prior probabilities, enabling a more scientific method to identify root causes rather than relying on guesswork.