Regression Modelling in Concrete and Material Science - CHAPTER FIVE Model Validation and Statistical Testing
CHAPTER FIVE
Model Validation and Statistical Testing
By Owus M. Ibearugbulem – Light for the Living
5.1 Overview
Constructing a regression model is only half the job. The real question is: Does the model truly represent reality? In this chapter, we focus on model validation — the process of checking how accurately a regression model describes experimental data. Ibearugbulem’s framework integrates several simple but powerful statistical tools to measure this accuracy and significance.
By the end of this chapter, you should be able to:
- Compute and interpret the Coefficient of Determination (\(R^2\)) and Adjusted \(R^2\).
- Calculate the Standard Error of Regression (SER).
- Perform Analysis of Variance (ANOVA) and compute the F-statistic.
- Decide whether a model is statistically significant and practically reliable.
5.2 Why Validate a Regression Model?
Even a model that fits a small dataset perfectly might fail on new data. Validation ensures that:
- The model explains most of the variation in the response variable.
- The predictors are genuinely relevant.
- Random errors are small and evenly distributed.
- The model can predict new cases accurately.
Without validation, a regression model is just a “best guess.”
5.3 The Key Validation Metrics
Ibearugbulem’s approach employs three principal validation measures and the ANOVA F-test.
| Statistic | Meaning | What It Measures |
|---|---|---|
| \(R^2\) | Coefficient of Determination | Overall fit of the model |
| \(R^2_{adj}\) | Adjusted R² | Fit adjusted for number of predictors |
| SER | Standard Error of Regression | Average prediction error |
| F | F-statistic | Overall model significance |
5.4 Coefficient of Determination (\(R^2\))
5.4.1 Definition
\[ R^2 = 1 - \frac{SSR}{TSS} \quad \text{(5.1)} \] Where:
\[ RSS = \sum (Y_j - \bar{Y})^2, \quad SSR = \sum (Y_j - Y_{mod,j})^2, \quad TSS = RSS + SSR \]
RSS = Regression Sum of Squares (explained variation)
SSR = Sum of Squared Residuals (unexplained variation)
TSS = Total Sum of Squares (total variation in \(Y\))
\(Y_j\) = measured value, \(\bar{Y}\) = mean of measured values, \(Y_{mod,j}\) = model predicted value.
5.4.2 Interpretation
- \(R^2 = 1\): perfect fit (model explains all variation).
- \(R^2 = 0\): model explains none of the variation.
- In engineering materials, \(R^2\) between 0.9 and 1.0 is considered excellent.
5.4.3 Worked Example
| Mix | Y (Measured) | Yₘ (Predicted) | Y − Ŷ | Y − Ȳ |
|---|---|---|---|---|
| 1 | 36.65 | 37.32 | 10.2 | -0.67 |
| 2 | 33.79 | 33.12 | 7.34 | 0.67 |
| 3 | 27.27 | 26.10 | 0.82 | 1.17 |
| 4 | 19.33 | 21.00 | -7.12 | -1.67 |
| 5 | 15.23 | 14.73 | -11.22 | 0.50 |
\(RSS = 535.54,\; SSR = 5.34,\; TSS = 540.88\) \[ R^2 = 1 - \frac{SSR}{TSS} = 1 - \frac{5.34}{540.88} = 0.9901 \] Interpretation: The model explains 99% of the variability in compressive strength — an excellent fit.
5.5 Adjusted Coefficient of Determination (\(R^2_{adj}\))
5.5.1 Formula
\[ R^2_{adj} = 1 - \frac{(n - 1)}{(n - k - 1)} (1 - R^2) \quad \text{(5.2)} \] Where \(n\) = number of observations and \(k\) = number of predictors.
5.5.3 Example
With \(n = 5,\; k = 3,\; R^2 = 0.9901\): \[ R^2_{adj} = 1 - \frac{(5 - 1)}{(5 - 3 - 1)} (1 - 0.9901) = 0.9604 \] Interpretation: Even after adjustment, 96% of the variation is still explained — the model remains highly reliable.
5.6 Standard Error of Regression (SER)
5.6.1 Definition
\[ SER = \sqrt{\frac{SSR}{n - k - 1}} \quad \text{(5.3)} \] A lower SER indicates better predictive accuracy.
5.6.2 Example
Using \(SSR = 5.34,\; n = 5,\; k = 3\): \[ SER = \sqrt{\frac{5.34}{(5 - 3 - 1)}} = \sqrt{5.34} = 2.31 \] Interpretation: On average, the model’s predictions differ from the laboratory results by ±2.31 MPa — a small error margin.
Think of SER like a fishing net. If the holes (errors) are small, you catch nearly all the fish (predictions). If the holes are large, many fish (true results) slip away. A small SER means your net — your model — fits the data tightly.
5.7 Analysis of Variance (ANOVA)
5.7.1 Concept
ANOVA divides total variation into two parts:
- Regression (explained) — variation explained by the model.
- Residual (unexplained) — random variation.
5.7.2 Equations
\[ TSS = RSS + SSR \quad \text{(5.4)} \]
| Source | Sum of Squares | Degrees of Freedom | Mean Square | Symbol |
|---|---|---|---|---|
| Regression | RSS | k | \(MSR = RSS/k\) | MSR |
| Residual (Error) | SSR | n − k − 1 | \(MSE = SSR/(n−k−1)\) | MSE |
| Total | TSS | n − 1 | – | – |
5.8 The F-Test for Model Significance
\[ F = \frac{MSR}{MSE} = \frac{RSS/k}{SSR/(n - k - 1)} = \frac{RSS(n - k - 1)}{SSRk} \quad \text{(5.5)} \] If \(F_{computed} > F_{critical}\), the model is significant — at least one predictor meaningfully affects the response.
5.8.1 Example
Given \(RSS = 535.54,\; SSR = 5.341,\; n = 5,\; k = 3\): \[ MSR = \frac{RSS}{k} = 178.51, \quad MSE = \frac{SSR}{n - k - 1} = 5.341 \] \[ F = \frac{MSR}{MSE} = 33.42 \] At 87% significance, \(F_{crit} = 31.53\). Since \(F > F_{crit}\), the model is statistically significant.
Imagine you’re a teacher comparing two students’ exam scripts: one filled with meaningful answers (model) and one filled with random guesses (error). The F-test tells you whether the “model’s script” really scores higher than chance — and in this example, it definitely does!
5.9 Summary of Validation Tools
| Metric | Formula | What It Indicates |
|---|---|---|
| \(R^2\) | \(1 - \frac{SSR}{TSS}\) | Proportion of total variation explained by model |
| \(R^2_{adj}\) | \(1 - \frac{(n-1)}{(n-k-1)}(1-R^2)\) | Adjusted for number of predictors |
| SER | \(\sqrt{\frac{SSR}{n-k-1}}\) | Average distance between predicted and observed values |
| F | \(\frac{RSS/k}{SSR/(n-k-1)}\) | Overall model significance |
5.10 Practical Steps in Classroom Validation
- Calculate all Y values (measured and predicted).
- Find residuals: \(e_i = Y_i − Y_{mod,i}\).
- Compute SSR, RSS, and TSS.
- Find \(R^2\), \(R^2_{adj}\), SER, and F.
- Compare \(F_{computed}\) with \(F_{critical}\).
- Interpret your results.
5.11 Summary
- Model validation ensures that the regression equation reliably predicts the response variable.
- Ibearugbulem’s validation suite uses \(R^2\), adjusted \(R^2\), SER, and F-tests to assess accuracy and soundness.
- A valid model should have \(R^2 > 0.9\), \(R^2_{adj}\) close to \(R^2\), small SER, and \(F_{computed} > F_{critical}\).
Test Your Understanding (Chapter 5)
- What does \(R^2\) represent in regression analysis?
- Explain why \(R^2_{adj}\) is sometimes more useful than \(R^2\).
- If SER = 10.5, what does this tell you about your model?
- In an F-test, what conclusion is drawn if \(F_{computed} < F_{critical}\)?
- Challenge: How is checking \(R^2\) like checking your exam score after studying a new topic?
Comments
Post a Comment