Tag
cross-entropy
- Dec 12, 2024 Rare-Event Estimation with Splitting and the Cross-Entropy Shift When the event of interest has probability a million to one, plain sampling sees nothing, and the standard tools bias the proposal or multiply the paths to reach the tail in bounded steps. scientific computing
- Mar 11, 2024 Proper Scoring Rules and Why They Cannot Be Gamed Why some losses reward honest probability forecasts while others reward sharpness alone, what decompositions reveal about calibration, and why the logarithmic score is the only local strictly proper rule. machine learning
- Feb 1, 2024 Why Perplexity Is Not Comparable Across Tokenizers Perplexity measures average surprise per token, so the vocabulary decides the unit. Converting to bits per byte removes the unit but not every confound. inference systems
- May 8, 2023 Logistic Regression as Conditional Bernoulli Likelihood How the sigmoid falls out of a log-odds parameterisation, why the negative log-likelihood is convex and unbounded in the separable case, and what the gradient actually computes. machine learning
- Apr 12, 2022 Softmax Cross-Entropy and Its Collapsed Gradient Why the gradient of the cross-entropy loss with respect to logits reduces to the prediction minus the target, what that saves computationally, and how to keep the exponentials from overflowing. deep learning