Tag
mutual-information
- Sep 15, 2026 InfoNCE and the Temperature Parameter How the contrastive objective turns similarity scores into a softmax over the batch, why the temperature controls effective negative count and gradient shape, and what breaks when it is set wrong. deep learning
- Mar 6, 2024 Shannon Entropy as the Expected Cost of an Outcome Why information is measured as the logarithm of reciprocal probability, how entropy averages that quantity over a source, and where the number stops describing anything real. computation theory
- Dec 12, 2023 Mutual Information and the Capacity of a Noisy Channel How the largest reliable transmission rate is defined as a maximum of mutual information, why the binary symmetric channel peaks at an interior input distribution, and what the capacity formula does not promise. computation theory