Transformers

A calculator multiplies two thousand-digit numbers and takes the remainder without thinking about it. Ask a neural network to do the same thing and it becomes an open research question, one that Terence Tao helped set as a competition. The answer has to be exact, because a remainder that is off by one is simply wrong. This is what I submitted to the SAIR Foundation’s Modular Arithmetic Challenge, why the obvious approach cannot work, and the one idea that made the difference.

The mathematics of the Transformer, worked through from the 2017 paper by Vaswani and colleagues. Scaled dot-product attention, why the scaling factor is there, multi-head attention, positional encoding, and the encoder and decoder stacks, each derived rather than asserted, for a reader who wants to understand the architecture rather than only use it.