Scenario
You built a regression model with R² = 0.87 and significant P-values. Your supervisor nodded and said, "Good." But when you used the model for prediction, you noticed that the predicted values were systematically lower in high-temperature regions.
What went wrong? R² was fine, P-values were fine, but the model assumptions were violated, and you didn't perform residual analysis.
What are Residuals
Residual = Actual Value - Predicted Value
Every data point has a residual, which represents the part not explained by the model. If the model were perfect, residuals should be random noise—without any pattern or form.
Residual analysis is about looking for patterns in residuals. Patterns indicate a problem with the model.
Four Essential Residual Plots
1. Residuals vs. Fitted Plot
X-axis: Model Predicted Values (Fitted Values)
Y-axis: Residuals
Ideal Scenario: Points are randomly scattered around 0, with no pattern.
Problem Signals:
- Fan-shaped spread (Heteroscedasticity): Residuals get larger with higher predicted values → consider taking the log of Y
- Curvilinear pattern (Non-linearity): → the model should include an X² term
- Grouping: → data might originate from different processes or batches
2. Normal Probability Plot of Residuals
Ideal Scenario: Points fall on a straight line.
Problem Signals:
- S-shaped curve: → residual distribution has heavy tails, possibly indicating outliers
- Two distinct lines: → data might be a mixture of two distributions
3. Residuals vs. Order Plot
X-axis: Data Collection Order
Y-axis: Residuals
Ideal Scenario: Random distribution.
Problem Signals:
- Trend: → indicates time-related effects (e.g., equipment wear, temperature drift)
- Periodicity: → shift effects or periodic interference
4. Histogram of Residuals
Ideal Scenario: Bell-shaped, symmetric distribution.
Problem Signals:
- Skewness: → might require transformation
- Bimodal: → data is a mixture of two populations
What to Do After Finding Issues
| Problem | Possible Solutions |
|---|---|
| Non-linearity | Add X² or interaction terms |
| Heteroscedasticity | Log transform Y or use weighted regression |
| Time Trend | Add "Time" as a covariate |
| Outliers | Confirm cause, then decide whether to keep or remove |
| Non-normality | Less impactful with large samples; consider non-parametric methods with small samples |
Residual Analysis vs. Model Metrics
| Metric | What it tells you | What it doesn't tell you |
|---|---|---|
| R² | Overall explanatory power | Whether model assumptions hold |
| P-value | Whether coefficients are significant | Whether residuals are random |
| Residual Analysis | Whether assumptions hold | Magnitude of explanatory power |
All three must be considered; none can be omitted.
Golden Quote
"R² tells you how good your model is; residual plots tell you if your model is lying. Only looking at R² is like only asking for the grade without checking if there was cheating."