Machine Learning · Cluster
ML Systems
Pipelines, drift, and reproducibility.
- 01 Training-Serving Skew and the Contract Between Features and Models How the same feature computed in two code paths diverges, why the split is structural rather than a bug, and which checks catch the drift before a model ships.
- 02 Feature Definitions as Versioned Artifacts Why a feature needs a name, a query, an owner, and a version, and how change control on feature definitions prevents silently retrained models.
- 03 Point-in-Time Correctness in Training Set Assembly Joins between labels and features must respect event time, not ingestion time; how to build a training table that never sees the future.
- 04 Detecting Distribution Drift in Production Data How to measure a shift in a feature or prediction distribution, which statistics survive high dimensionality, and how alert thresholds avoid false alarms.
- 05 Data Validation at Pipeline Boundaries Schema checks, value constraints, and distribution assertions that stop corrupt data before it reaches training, and what each class of check costs.
- 06 Reproducing Training Runs from Recorded Metadata What a training run must record so that a second execution yields the same weights, and where nondeterminism survives even a perfect record.
- 07 Offline Evaluation Leakage and the Metrics That Hide It How random splits, duplicated entities, and temporal leakage inflate offline metrics, and which split design gives an estimate that predicts deployment.
- 08 How Automated Retraining Loops Close Without Oscillating Retraining triggers, shadow evaluation, and rollback criteria: the control loop that keeps a served model current without thrashing or self-confirmation.
- 09 Model Registries and Promotion Gates Between Training and Traffic What a registry records about an artifact, how staging transitions are gated on evidence, and why the serving path must resolve versions rather than names.
- 10 Scheduling Batch Training Against Shared Accelerator Capacity Gang scheduling, quota partitioning, and preemption for training jobs: why a GPU job is not a long-running process, and what fair sharing costs.