Machine Learning · Cluster
Model Evaluation
Validation schemes and metric selection.
- 01 Why a Held-Out Test Set Cannot Be Reused What a test-set score actually estimates, how a second look at the same rows biases it upward, and the leakage paths that corrupt the number before any model is fitted.
- 02 Stratified k-Fold Cross-Validation and the Variance of the Estimate How k-fold partitioning produces a spread estimate, why the folds are not independent, and what stratification actually removes from the variance.
- 03 Selecting a Decision Threshold from the Confusion Matrix Why a classifier produces a score rather than a label, how the threshold trades false positives against false negatives, and what determines where it belongs.
- 04 Macro, Micro, and Weighted Averaging in Multiclass Metrics How the three averaging modes weight each class, why micro-averaged F1 equals accuracy in single-label problems, and which one answers which question.
- 05 Blocked and Rolling-Origin Validation for Time-Ordered Data Why random folds fail on time series, how expanding and sliding origins differ in what they measure, and where the gap between folds must be inserted.
- 06 Grouped Splits When Observations Are Not Independent How shared subjects across rows inflate a random split, what group-aware partitioning guarantees, and how to choose the grouping key when several exist.
- 07 ROC-AUC and PR-AUC Under Class Imbalance What each area actually integrates, why ROC-AUC stays flat as positives become rare, and the conditions under which the two order two models differently.
- 08 Calibrating Scores Before Choosing an Operating Point How to measure whether a score means what it claims, why Platt scaling and isotonic regression behave differently at the ends, and what calibration cannot fix.
- 09 Nested Cross-Validation and the Optimism of Tuned Scores Why a cross-validated score from a tuned procedure is biased, how the inner and outer loops separate selection from estimation, and what the procedure costs.
- 10 Comparing Two Classifiers with Paired Tests on the Same Rows How to test whether an observed difference in error is larger than fold noise, why the resampled t-test fails, and what the corrected alternatives assume.