Section 3.4 Generative model architecture

Decoding, context and inference cost

At every step, the model turns the last position’s representation into scores, called logits, for the tokens in its vocabulary. Greedy decoding chooses the largest score. Sampling draws a token from a distribution. The selected token joins the input before the next step.

Temperature and probability mass

With strictly positive temperature T, divide the logits by T before softmax. A small value concentrates probability on the highest scores. Top-p sampling sorts tokens by probability and keeps the smallest prefix that reaches the chosen mass, then renormalizes. With probabilities 0.60, 0.25, 0.10 and 0.05, a cutoff of 0.80 keeps two options. Their new probabilities are approximately 0.706 and 0.294. This procedure changes output diversity. The probabilities describe text continuations and inherit the limitations of the underlying training.

Four probabilities sum to one. The cumulative mass reaches 0.85 after the first two options.
Figure 3.4. Before renormalization, the two leading options total 0.85 and remain candidates.

Fit information into the context window

Instructions, conversation history, reference passages and the reply share finite capacity. Even when all the material fits, a model can overlook a crucial detail. Time to first token includes processing the input text. Sequential generation adds latency for each output token. A key-value cache keeps computations from previous positions so they are not repeated at every step, using additional memory. In a document task, dropping irrelevant passages can improve both cost and the visibility of the evidence.