-
How Much Softmax Do You Need in LLMs?
What a pretrained LLM needs from softmax, a calibration-free FlashAttention-4 kernel, and what changes when you train with a coarse softmax.
What a pretrained LLM needs from softmax, a calibration-free FlashAttention-4 kernel, and what changes when you train with a coarse softmax.