Machine Learning · Cluster
Tree Ensembles
Bagging, boosting, and split criteria.
- 01 How CART Scans Sorted Feature Values for the Best Split Threshold The greedy split search reduces to a single sorted pass per feature, with incremental class counts turning an apparent quadratic problem into a linear scan.
- 02 Gini, Entropy, and Why Misclassification Error Cannot Grow a Tree Three impurity functions look interchangeable, but only the strictly concave ones produce a useful ranking of splits. A worked partition shows where the third fails.
- 03 Cost-Complexity Pruning and the Weakest-Link Sequence How a maximal tree is collapsed into a nested family of subtrees indexed by one penalty parameter, and why the sequence makes cross-validation cheap.
- 04 Bagging: Bootstrap Resampling and the Variance It Removes Averaging models trained on bootstrap resamples reduces variance only as far as the models are uncorrelated, which explains both the gains and the cases where there are none.
- 05 Split-Time Feature Subsampling in Random Forests Restricting the candidate columns at every split lowers the correlation between trees, and the strength-correlation trade explains why the default is neither one feature nor all of them.
- 06 Gradient Boosting as Gradient Descent in Function Space Each boosting round fits a small tree to the negative gradient of the loss at the current predictions, which makes the algorithm a steepest-descent step whose direction comes from a restricted model class.
- 07 Second-Order Boosting and the Regularized Split Gain A quadratic expansion of the loss gives closed-form leaf weights from gradient and curvature sums, turning each split into a comparison of two ratios and making regularization part of the objective.
- 08 Histogram Split Finding and the Weighted Quantile Sketch Binning a feature into a few hundred buckets turns repeated per-node sorts into linear passes, and choosing the boundaries by weighted quantiles keeps the buckets where the data actually is.
- 09 Monotonic and Interaction Constraints in Boosted Ensembles Bounding each node interval and restricting which features may share a path turn domain assumptions into structural properties of every tree, at a measurable cost in fit.
- 10 Why Piecewise-Constant Tree Ensembles Cannot Extrapolate Beyond the largest value seen in training, every threshold test returns the same answer, so the prediction freezes. The consequence and the partial remedies follow from the shape of the fitted function.