Section 3.2 Generative model architecture

Causal attention and the transformer

Self-attention builds a weighted mixture of representations from different positions. At each position, learned transformations produce a query vector q, a key k and a value v. A query is compared with keys through dot products. The scores, scaled by √dₖ, pass into softmax, which gives positive weights summing to 1 over the permitted positions. The output is a weighted sum of values.

sⱼ = (q · kⱼ) / √dₖ

αⱼ = exp(sⱼ) / Σᵢ exp(sᵢ)

output = Σⱼ αⱼvⱼ

A calculation you can reproduce

For query q = (1, 0), keys (1, 0) and (0, 1), and key dimension dₖ equal to 2, the scaled scores are 1/√2 and 0. The softmax weights are approximately 0.670 and 0.330. With values (2, 0) and (0, 2), the output is approximately (1.340, 0.660). Keys help select contributions. Values carry the numbers to combine. A real model learns these transformations from data.

Five-position causal grid. A row can read its diagonal and the cells to its left. Cells to its right are masked.
Figure 3.2. Read each row as a query position and each column as a key position. The shaded pattern marks access. It does not show attention weights.

Why mask future positions?

In an autoregressive decoder, a position may use only its own input and earlier inputs when predicting the following token. The mask prevents reading the answer during training. A position mechanism also makes token order available. A transformer block combines attention heads with position-wise transformations, residual connections and normalization. Vaswani et al. describe the original architecture in an encoder–decoder setting.