Topic domain
Machine Learning
Statistical learning from data, and its evaluation.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Regression, classification, and loss design.
- Least Squares as Orthogonal Projection onto a Design Matrix
- Logistic Regression as Conditional Bernoulli Likelihood
- Ridge Shrinkage in Ill-Conditioned Design Matrices
- Squared, Absolute, and Quantile Loss Compared by Their Gradients
- Class Imbalance, Class Priors, and Cost-Sensitive Thresholds
- Newton and IRLS Fitting for Logistic Regression
Validation schemes and metric selection.
- Why a Held-Out Test Set Cannot Be Reused
- Stratified k-Fold Cross-Validation and the Variance of the Estimate
- Selecting a Decision Threshold from the Confusion Matrix
- Macro, Micro, and Weighted Averaging in Multiclass Metrics
- Blocked and Rolling-Origin Validation for Time-Ordered Data
- Grouped Splits When Observations Are Not Independent
Representation, scaling, and leakage.
- Feature Scaling and the Conditioning of Gradient Descent
- One-Hot Encoding as Geometry and the Dummy Variable Trap
- How a Feature Becomes a Leak
- Out-of-Fold Target Encoding for High-Cardinality Categoricals
- Transformer State and the Fold Boundary
- Mean Imputation and the Variance It Removes
Bagging, boosting, and split criteria.
- How CART Scans Sorted Feature Values for the Best Split Threshold
- Gini, Entropy, and Why Misclassification Error Cannot Grow a Tree
- Cost-Complexity Pruning and the Weakest-Link Sequence
- Bagging: Bootstrap Resampling and the Variance It Removes
- Split-Time Feature Subsampling in Random Forests
- Gradient Boosting as Gradient Descent in Function Space
Clustering, dimensionality reduction, and density.
- Lloyd's Algorithm and the Local Minima of k-Means
- What a Distance Function Decides About Cluster Shape
- Agglomerative Clustering and the Lance-Williams Recurrence
- Gaussian Mixture Models and the EM Lower Bound
- DBSCAN, Density Reachability, and the Parameter Cliff
- Principal Component Analysis Through the Singular Value Decomposition
Graphical models and Bayesian inference.
- Conjugate Priors and the Arithmetic of Posterior Updating
- Conditional Independence and the Factorization of Joint Distributions
- Directed Graphical Models: D-Separation and the Markov Blanket
- Exact Inference by Variable Elimination and the Cost of Treewidth
- Markov Random Fields and the Intractability of the Partition Function
- Belief Propagation on Trees and Why Loopy Message Passing Fails
Optimization
10Gradients, constraints, and convergence.
- Step Sizes and the Descent Lemma in Gradient Descent
- Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent
- Momentum and Nesterov Acceleration in Gradient Descent
- Adaptive Step Sizes: AdaGrad, RMSProp, and Adam
- Lagrange Duality and the KKT Conditions in Constrained Learning
- Projected and Proximal Gradient Methods for Composite Objectives
ML Systems
10Pipelines, drift, and reproducibility.
- Training-Serving Skew and the Contract Between Features and Models
- Feature Definitions as Versioned Artifacts
- Point-in-Time Correctness in Training Set Assembly
- Detecting Distribution Drift in Production Data
- Data Validation at Pipeline Boundaries
- Reproducing Training Runs from Recorded Metadata
All Machine Learning articles
- Oct 21, 2026 Directed Graphical Models: D-Separation and the Markov Blanket How a directed acyclic graph selects a factorization, why three path shapes reduce to one blocking rule, and what the Markov blanket lets an algorithm ignore.
- Oct 15, 2026 Variational Inference and the Evidence Lower Bound Why maximizing a lower bound replaces integrating a posterior, how mean-field updates are derived coordinate by coordinate, and what the bound systematically understates.
- Oct 8, 2026 Point-in-Time Correctness in Training Set Assembly Joins between labels and features must respect event time, not ingestion time; how to build a training table that never sees the future.
- Oct 7, 2026 Conjugate Priors and the Arithmetic of Posterior Updating How a likelihood and a prior combine into a posterior in closed form, what that closure preserves under sequential updating, and where the closed form stops being available.
- Sep 16, 2026 Kernel Density Estimation and the Bandwidth Trade-off Placing a kernel on every observation gives a density estimate with one smoothing parameter, and the rate at which that parameter must shrink with sample size explains why the method becomes almost unusable in high dimensions.
- Sep 10, 2026 Why a Held-Out Test Set Cannot Be Reused What a test-set score actually estimates, how a second look at the same rows biases it upward, and the leakage paths that corrupt the number before any model is fitted.
- Sep 5, 2026 Adaptive Step Sizes: AdaGrad, RMSProp, and Adam How per-coordinate accumulators of squared gradients rescale the update, why the resulting methods needed repair to keep their convergence guarantees, and what the state costs.
- Aug 27, 2026 Squared, Absolute, and Quantile Loss Compared by Their Gradients How the shape of a regression loss decides which observations move the fit, why the absolute loss has no derivative at zero, and what the pinball loss estimates instead of a mean.
- Aug 21, 2026 Scheduling Batch Training Against Shared Accelerator Capacity Gang scheduling, quota partitioning, and preemption for training jobs: why a GPU job is not a long-running process, and what fair sharing costs.
- Aug 19, 2026 What a Distance Function Decides About Cluster Shape Squared Euclidean, cosine, Mahalanobis, and set distances each impose a different notion of similarity, and the choice fixes which clusters are available before any algorithm runs.
- Aug 19, 2026 Why t-SNE Preserves Neighborhoods but Distorts Distances The embedding minimizes one asymmetric divergence between two similarity distributions, so it pulls true neighbours together while almost ignoring far pairs, which makes cluster sizes and inter-cluster gaps artefacts of the optimization.
- Jul 29, 2026 Split-Time Feature Subsampling in Random Forests Restricting the candidate columns at every split lowers the correlation between trees, and the strength-correlation trade explains why the default is neither one feature nor all of them.
- Jul 24, 2026 Model Registries and Promotion Gates Between Training and Traffic What a registry records about an artifact, how staging transitions are gated on evidence, and why the serving path must resolve versions rather than names.
- Jul 21, 2026 Gradient Boosting as Gradient Descent in Function Space Each boosting round fits a small tree to the negative gradient of the loss at the current predictions, which makes the algorithm a steepest-descent step whose direction comes from a restricted model class.
- Jul 5, 2026 Feature Scaling and the Conditioning of Gradient Descent How per-feature magnitude sets the condition number of a loss surface, why that number fixes the iteration count of first-order methods, and what each scaler costs.
- Jun 18, 2026 How a Feature Becomes a Leak Leakage defined as a dependency between the training representation and the evaluation labels, the three mechanisms that create it, and why it inflates validation without touching training error.
- Jun 16, 2026 Cost-Complexity Pruning and the Weakest-Link Sequence How a maximal tree is collapsed into a nested family of subtrees indexed by one penalty parameter, and why the sequence makes cross-validation cheap.
- Jun 12, 2026 Principal Component Analysis Through the Singular Value Decomposition Maximizing projected variance, minimizing reconstruction error, and decorrelating features all select the same subspace, and the SVD computes it without ever forming the covariance matrix.
- Jun 6, 2026 Histogram Split Finding and the Weighted Quantile Sketch Binning a feature into a few hundred buckets turns repeated per-node sorts into linear passes, and choosing the boundaries by weighted quantiles keeps the buckets where the data actually is.
- May 6, 2026 Data Validation at Pipeline Boundaries Schema checks, value constraints, and distribution assertions that stop corrupt data before it reaches training, and what each class of check costs.
- May 4, 2026 Variance Reduction in Stochastic Gradient Descent: SVRG and SAGA How control variates make the stochastic gradient noise vanish near the solution, why that restores a linear rate for finite sums, and what the stored state and epoch length cost.
- Apr 3, 2026 Metropolis-Hastings: Detailed Balance and Proposal Design How a Markov chain is built so that an unnormalized density becomes its stationary distribution, what the acceptance ratio compensates for, and when a correctly implemented sampler still mixes badly.
- Feb 3, 2026 Conditional Independence and the Factorization of Joint Distributions Why a joint distribution over many variables cannot be stored as a table, how independence assertions turn it into a product of small factors, and what the factored form can never represent.
- Jan 6, 2026 How CART Scans Sorted Feature Values for the Best Split Threshold The greedy split search reduces to a single sorted pass per feature, with incremental class counts turning an apparent quadratic problem into a linear scan.
- Dec 22, 2025 Reproducing Training Runs from Recorded Metadata What a training run must record so that a second execution yields the same weights, and where nondeterminism survives even a perfect record.
- Dec 13, 2025 Structure Learning and the Limits of Markov Equivalence How independence tests and scoring functions recover a graph from data, why observation alone cannot orient every edge, and what the search has to assume to terminate.
- Dec 8, 2025 Expectation-Maximization as Inference with Latent Variables Why the log outside a sum over hidden variables blocks direct maximization, how an exact E step makes a lower bound tight enough to guarantee progress, and where the fixed points end up.
- Nov 4, 2025 Class Imbalance, Class Priors, and Cost-Sensitive Thresholds Why resampling and thresholding both leave the fitted probabilities unchanged or wrong, how a prior shift translates into an intercept adjustment, and where the expected-cost optimum sits.
- Nov 4, 2025 Selecting a Decision Threshold from the Confusion Matrix Why a classifier produces a score rather than a label, how the threshold trades false positives against false negatives, and what determines where it belongs.
- Oct 13, 2025 Transformer State and the Fold Boundary Why every fitted preprocessing object must live inside the cross-validation loop, which statistics actually matter, and how refitting at the end reintroduces the leak.
- Oct 2, 2025 How Automated Retraining Loops Close Without Oscillating Retraining triggers, shadow evaluation, and rollback criteria: the control loop that keeps a served model current without thrashing or self-confirmation.
- Sep 26, 2025 Group and Duplicate Leakage in Cross-Validation Splits Why row-level splitting fails when rows are not independent, how grouped and blocked folds restore the estimate, and what the fold count costs in group imbalance.
- Sep 25, 2025 Lagrange Duality and the KKT Conditions in Constrained Learning How multipliers convert constrained problems into stationary points of an unconstrained function, when the dual problem has the same value as the primal, and where these conditions stop being sufficient.
- Sep 3, 2025 Agglomerative Clustering and the Lance-Williams Recurrence Bottom-up merging needs one rule for the distance between two groups of points; the Lance-Williams update is the shared arithmetic behind single, complete, average, and Ward linkage, and it decides which hierarchies are order preserving.
- Sep 1, 2025 Training-Serving Skew and the Contract Between Features and Models How the same feature computed in two code paths diverges, why the split is structural rather than a bug, and which checks catch the drift before a model ships.
- Aug 28, 2025 Bagging: Bootstrap Resampling and the Variance It Removes Averaging models trained on bootstrap resamples reduces variance only as far as the models are uncorrelated, which explains both the gains and the cases where there are none.
- Aug 24, 2025 Lloyd's Algorithm and the Local Minima of k-Means The assignment and update steps are two exact minimizations of the same objective, which is why the loop always terminates, why it stops at a local minimum, and what seeding changes about the answer.
- Aug 22, 2025 Least Squares as Orthogonal Projection onto a Design Matrix Why the normal equations are a geometric condition rather than an algebraic trick, what rank deficiency does to the answer, and why forming the Gram matrix costs half the digits.
- Aug 15, 2025 Step Sizes and the Descent Lemma in Gradient Descent How the Lipschitz constant of the gradient bounds a usable step size, why 1/L is the natural choice for quadratic objectives, and which constants the convergence rates hide.
- Aug 2, 2025 Spectral Clustering and the Relaxation of Normalized Cut Turning points into a weighted graph, writing the partition cost as a cut, and relaxing the discrete indicator vector into an eigenvector problem explains why the method recovers non-convex groups and why the affinity scale decides the answer.
- Jul 24, 2025 Why Piecewise-Constant Tree Ensembles Cannot Extrapolate Beyond the largest value seen in training, every threshold test returns the same answer, so the prediction freezes. The consequence and the partial remedies follow from the shape of the fitted function.
- Jul 21, 2025 Feature Hashing and the Collision Bound That Sets Its Width Why signed hashing leaves inner products unbiased, how many collisions a fixed width produces, and the error term that appears in place of a vocabulary lookup.
- Jul 3, 2025 Momentum and Nesterov Acceleration in Gradient Descent How velocity terms turn the gradient sequence into a filtered signal, why the quadratic case admits a closed-form optimal damping, and what breaks when gradients are noisy or the objective is not convex.
- May 30, 2025 Cluster Validity, Stability, and Choosing the Number of Clusters Internal indices measure geometry that a clustering may not assume, stability is not a consistent selector, and chance-corrected agreement between resampled partitions answers a narrower question than it appears to.
- May 26, 2025 One-Hot Encoding as Geometry and the Dummy Variable Trap Why one-hot columns span a subspace whose dimensions are the category count minus one, what collinearity the redundant column introduces, and when the penalty makes it harmless.
- May 4, 2025 ROC-AUC and PR-AUC Under Class Imbalance What each area actually integrates, why ROC-AUC stays flat as positives become rare, and the conditions under which the two order two models differently.
- Apr 29, 2025 Mean Imputation and the Variance It Removes Substituting the column mean for a missing value preserves the mean exactly and shrinks the variance and the covariance, changing coefficients, standard errors, and fit.
- Apr 5, 2025 Parallel and Asynchronous SGD: Staleness and Convergence Bounds How splitting a gradient computation across workers changes the update rule, why delayed gradients degrade the guarantee in proportion to the delay, and what communication efficiency costs in accuracy.
- Dec 13, 2024 Feature Definitions as Versioned Artifacts Why a feature needs a name, a query, an owner, and a version, and how change control on feature definitions prevents silently retrained models.
- Dec 5, 2024 Link Functions, Variance Functions, and Poisson Regression with Exposure How the log link keeps fitted rates positive and turns coefficients into multiplicative effects, why the mean-variance relation is a modelling assumption, and what an offset does that a predictor cannot.
- Oct 15, 2024 Gaussian Mixture Models and the EM Lower Bound Soft cluster membership follows from a latent-variable model whose log-likelihood has no closed-form maximizer; the EM iteration maximizes a lower bound that is tight at the posterior, and it can also drive components to degenerate solutions.
- Oct 10, 2024 Out-of-Fold Target Encoding for High-Cardinality Categoricals How smoothing trades bias for variance in target encoding, why the encoding must be computed out of fold, and what the left-out fold estimate costs in noise.
- Sep 28, 2024 Detecting Distribution Drift in Production Data How to measure a shift in a feature or prediction distribution, which statistics survive high dimensionality, and how alert thresholds avoid false alarms.
- Aug 9, 2024 Belief Propagation on Trees and Why Loopy Message Passing Fails How sum-product messages re-derive exact marginals on a tree, and what a fixed point on a graph with cycles does and does not guarantee.
- Jul 10, 2024 Gini, Entropy, and Why Misclassification Error Cannot Grow a Tree Three impurity functions look interchangeable, but only the strictly concave ones produce a useful ranking of splits. A worked partition shows where the third fails.
- Jun 19, 2024 Escaping Saddle Points in Nonconvex Optimization Why a vanishing gradient is not evidence of a minimum in high dimensions, how negative curvature is detected and followed, and what the escape guarantees cost per iteration.
- Jun 18, 2024 Exact Inference by Variable Elimination and the Cost of Treewidth How factors are multiplied and summed out, why the elimination order determines the size of every intermediate table, and why exact inference is exponential in treewidth.
- May 7, 2024 Markov Random Fields and the Intractability of the Partition Function Why undirected models factorize over cliques, how the normalizing sum becomes a partition function, and why that single constant governs both inference and learning.
- May 2, 2024 Softmax Coupling versus One-vs-Rest Probabilities What the normalising constant does to multiclass estimates, why one-vs-rest scores need not sum to one, and how the two parameterisations differ under class imbalance.
- Apr 28, 2024 Newton and IRLS Fitting for Logistic Regression Why logistic regression is fitted by repeated weighted least squares, how the Hessian decides the step, and what happens when the weights collapse under separation.
- Apr 10, 2024 Macro, Micro, and Weighted Averaging in Multiclass Metrics How the three averaging modes weight each class, why micro-averaged F1 equals accuracy in single-label problems, and which one answers which question.
- Mar 11, 2024 Proper Scoring Rules and Why They Cannot Be Gamed Why some losses reward honest probability forecasts while others reward sharpness alone, what decompositions reveal about calibration, and why the logarithmic score is the only local strictly proper rule.
- Mar 5, 2024 Offline Evaluation Leakage and the Metrics That Hide It How random splits, duplicated entities, and temporal leakage inflate offline metrics, and which split design gives an estimate that predicts deployment.
- Feb 29, 2024 Hinge Loss, Maximum Margin, and the Points That Decide the Boundary Why the hinge loss produces a sparse solution supported only on near-boundary points, what the margin constraint actually constrains, and how C trades margin width against violations.
- Feb 14, 2024 Second-Order Boosting and the Regularized Split Gain A quadratic expansion of the loss gives closed-form leaf weights from gradient and curvature sums, turning each split into a comparison of two ratios and making regularization part of the objective.
- Feb 10, 2024 Quasi-Newton Methods and the L-BFGS Two-Loop Recursion How secant equations build a Hessian approximation from successive gradient differences, why the limited-memory form stores only a few vector pairs, and how line searches keep the approximation positive definite.
- Dec 29, 2023 Calibrating Scores Before Choosing an Operating Point How to measure whether a score means what it claims, why Platt scaling and isotonic regression behave differently at the ends, and what calibration cannot fix.
- Sep 15, 2023 Nested Cross-Validation and the Optimism of Tuned Scores Why a cross-validated score from a tuned procedure is biased, how the inner and outer loops separate selection from estimation, and what the procedure costs.
- Aug 24, 2023 Variance-Stabilising Transforms for Counts and Proportions Why the log, square root, and arcsine transforms equalise variance across a mean-variance relation, and what changes when the counts include zeros.
- Aug 15, 2023 Comparing Two Classifiers with Paired Tests on the Same Rows How to test whether an observed difference in error is larger than fold noise, why the resampled t-test fails, and what the corrected alternatives assume.
- Jul 6, 2023 Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent Why sampling a subset of the data makes the gradient unbiased but noisy, how that noise sets a floor under constant step sizes, and what step schedules can and cannot fix.
- Jun 20, 2023 Monotonic and Interaction Constraints in Boosted Ensembles Bounding each node interval and restricting which features may share a path turn domain assumptions into structural properties of every tree, at a measurable cost in fit.
- May 30, 2023 Point-in-Time Feature Assembly and the As-Of Join How to compute a feature as it stood at a labelled event, why an as-of join is not a filtered join, and where late-arriving records break the reconstruction.
- May 8, 2023 Logistic Regression as Conditional Bernoulli Likelihood How the sigmoid falls out of a log-odds parameterisation, why the negative log-likelihood is convex and unbounded in the separable case, and what the gradient actually computes.
- Apr 10, 2023 Ridge Shrinkage in Ill-Conditioned Design Matrices How adding a multiple of the identity to the Gram matrix reweights singular directions, what the bias-variance trade actually buys, and why the penalty must not include the intercept.
- Mar 12, 2023 Projected and Proximal Gradient Methods for Composite Objectives How constraint sets and non-differentiable penalties are handled by composing a gradient step with a projection or a proximal operator, and what the convergence rates assume.
- Jan 10, 2023 Stratified k-Fold Cross-Validation and the Variance of the Estimate How k-fold partitioning produces a spread estimate, why the folds are not independent, and what stratification actually removes from the variance.
- Jan 5, 2023 Blocked and Rolling-Origin Validation for Time-Ordered Data Why random folds fail on time series, how expanding and sliding origins differ in what they measure, and where the gap between folds must be inserted.
- Jul 21, 2022 DBSCAN, Density Reachability, and the Parameter Cliff A cluster as a maximal set of density-connected points needs only a radius and a count, produces non-convex shapes and explicit noise, and is order sensitive only at the boundary of a cluster.
- May 12, 2022 Grouped Splits When Observations Are Not Independent How shared subjects across rows inflate a random split, what group-aware partitioning guarantees, and how to choose the grouping key when several exist.