Tag
softmax
- Oct 12, 2026 Causal Masking, Additive Mask Terms, and Where the Mask Enters the Softmax Why a causal constraint is expressed as negative infinity added to the logits, how padding and causality combine in one layer, and which arithmetic the mask must precede. deep learning
- May 21, 2026 Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension Where the 1/sqrt(d_k) factor comes from, what quantity it holds constant as width changes, and how a saturated softmax destroys the gradient reaching queries and keys. deep learning
- May 2, 2024 Softmax Coupling versus One-vs-Rest Probabilities What the normalising constant does to multiclass estimates, why one-vs-rest scores need not sum to one, and how the two parameterisations differ under class imbalance. machine learning
- Apr 12, 2022 Softmax Cross-Entropy and Its Collapsed Gradient Why the gradient of the cross-entropy loss with respect to logits reduces to the prediction minus the target, what that saves computationally, and how to keep the exponentials from overflowing. deep learning