Deep Learning · Cluster
Representation Learning
Embeddings, contrastive methods, and transfer.
- 01 Embedding Layers as Learned Lookup Tables An embedding table is a matrix indexed by token id whose rows receive sparse gradients; its memory arithmetic, initialization, and weight tying decide what a model can learn from rare inputs.
- 02 Distance Concentration and Hubness in Embedding Spaces Why nearest-neighbour search degrades in high dimensions, how a few points become hubs that absorb most queries, and what those effects imply for measuring similarity between learned vectors.
- 03 InfoNCE and the Temperature Parameter How the contrastive objective turns similarity scores into a softmax over the batch, why the temperature controls effective negative count and gradient shape, and what breaks when it is set wrong.
- 04 Data Augmentation as an Invariance Specification Augmentations declare which transformations should not change a representation; the choice fixes what the embedding can ignore, and a mismatched set teaches the model to discard signal it needs later.
- 05 Momentum Encoders and Stop-Gradients Why contrastive and self-distillation methods train one branch with gradients and update the other by exponential moving average, and what the asymmetry buys in stability and target quality.
- 06 False Negatives and Batch Composition Every off-diagonal entry in a contrastive batch is treated as a negative, including entries that are semantically identical; the resulting pressure shapes the space and is controlled by what goes into the batch.
- 07 Linear Probing and Fine-Tuning as Different Measurements A frozen-feature linear probe and full fine-tuning answer different questions about a pretrained encoder, and the gap between their results is itself the informative quantity.
- 08 Alignment and Uniformity as Separate Objectives Contrastive losses can be decomposed into a term that pulls positive pairs together and a term that spreads all representations apart, and the two terms trade against each other in measurable ways.
- 09 Collapse and Redundancy Reduction in Self-Supervised Training Why self-supervised objectives tend toward constant or low-rank outputs, how whitening and decorrelation terms restore full rank, and what each approach costs in compute and stability.
- 10 Cross-Modal Alignment and the Modality Gap Contrastive training can put matching images and text close together while the two modalities still occupy separate regions of the space; what produces that gap and what it costs for retrieval and composition.