“In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and each token is converted into a vector through an embedding layer.”
The quiet idea behind almost every AI product today. Attention let models look at the whole context at once instead of word by word, and that one change made scaling work.
