hidden · · 6 min read
One token, zero materialized logits
FlashSampling fuses exact categorical sampling into the LM head. Its authors report zero logits materialization in HBM and up to 10% faster decoding.
Writing · Tagged
AI
0 articles
hidden · · 5 min read
Softmax is not confidence
A classifier can print 0.93 without being correct 93% of the time. Softmax normalizes logits. Calibration is a separate empirical claim.
hidden · · 6 min read
Who actually chooses the next token?
The network produces logits. Temperature, truncation, and a random draw turn those logits into the token users see.
hidden · · 12 min read
Matrix calculus for deep learning, without the fog
A geometric and probabilistic refresher on the matrix calculus that actually matters in deep learning: gradients, Jacobians, affine maps, chain rules, and the softmax-cross-entropy shortcut.
hidden · · 6 min read
The draft model never gets the last word
The accept rule min(1, p/q) plus a residual draw keeps every emitted token distributed exactly as the target model would have chosen it, whatever the draft looks like, for 2.6x to 3.4x faster decoding on T5-XXL.
hidden · · 7 min read
The posterior is expensive either way
Knotkit can sample a finite list in one call. A VAE pays during training for a fast approximation. A Markov chain pays during inference for asymptotically exact samples.
No articles with this tag yet.