Section 3.2 Generative model architecture
Causal attention and the transformer
Self-attention builds a weighted mixture of representations from different positions. At each position, learned transformations produce a query vector q, a key k and a value v. A query is compared with keys through dot products. The scores, scaled by √dₖ, pass into softmax, which gives positive weights summing to 1 over the permitted positions. The output is a weighted sum of values.
sⱼ = (q · kⱼ) / √dₖ
αⱼ = exp(sⱼ) / Σᵢ exp(sᵢ)
output = Σⱼ αⱼvⱼ
A calculation you can reproduce
For query q = (1, 0), keys (1, 0) and (0, 1), and key dimension dₖ equal to 2, the scaled scores are 1/√2 and 0. The softmax weights are approximately 0.670 and 0.330. With values (2, 0) and (0, 2), the output is approximately (1.340, 0.660). Keys help select contributions. Values carry the numbers to combine. A real model learns these transformations from data.
Why mask future positions?
In an autoregressive decoder, a position may use only its own input and earlier inputs when predicting the following token. The mask prevents reading the answer during training. A position mechanism also makes token order available. A transformer block combines attention heads with position-wise transformations, residual connections and normalization. Vaswani et al. describe the original architecture in an encoder–decoder setting.