Capabilities · Intro
Transformers and Attention
This site set itself one rule before writing its first line: never redo, worse, what already exists in superb form. This page is the first to apply it.
The Transformer still needed explaining. It is the architecture behind nearly everything discussed here: ChatGPT, Claude, Gemini, image generators, and almost every system this site dissects story after story. We had started sketching our diagrams. Then we reread the Financial Times piece, and put ours away.
In September 2023, the FT’s visual storytelling team published “Generative AI exists because of the transformer”[1]. You scroll, and the architecture unfolds in order: words become tokens, tokens become numbers, the numbers weigh one another, and meaning comes out of that weighing. Nearly three years on, it remains the finest way in we know. It is free to read, and worth every one of your minutes.
──────[ the referral ]──────
“Generative AI exists because of the transformer”
Financial Times · visual storytelling · September 2023 · free to read
And if you want to go back to the source of the source, eight pages are enough.
──────[ attention is all you need · 2017 ]──────
June 2017. Eight Google researchers publish “Attention Is All You Need”[2], the paper that replaces word-by-word reading with a single principle: let every word look at every other word, and learn which ones matter. The diagram above replays that gaze, on the canonical example popularized alongside the paper[3]. Almost all of modern AI descends from those eight pages.
The day someone does better than the FT, we will redo this page. Until then, it stays short on purpose: that is its way of being honest. And to understand why making these Transformers bigger made them so capable, the natural next step is our story on scaling laws.
Sources & further reading
- 1. Financial Times (Visual Storytelling), « Generative AI exists because of the transformer », septembre 2023
- 2. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, « Attention Is All You Need », NeurIPS 2017 (arXiv:1706.03762)
- 3. J. Uszkoreit, « Transformer: A Novel Neural Network Architecture for Language Understanding », Google Research Blog, août 2017