Section 3.3 Generative model architecture

Pretraining, adaptation and post-training

For tokenized text t₁, …, tₙ, an autoregressive objective asks each position to predict the next token. During training, the observed continuation supplies the target. Negative log-likelihood strongly penalizes targets assigned very small probabilities. A probability of 0.1 costs approximately 2.303 nats, compared with 0.223 at 0.8.

What does perplexity measure?

Perplexity is the exponential of average loss per token when the loss uses the natural logarithm. A model that always spreads its probability uniformly across 10 possibilities has perplexity 10 on text whose targets belong to that vocabulary. Compare perplexities with the same corpus, segmentation and evaluation protocol. This metric measures prediction of observed text. A task such as providing accurate citations needs its own evaluation.

A corpus supplies pretraining, demonstrations refine instruction following, and preferences can shape replies.
Figure 3.3. One possible training sequence. Post-training data and methods vary across models.

Change the model or change its request

Supervised fine-tuning on demonstrations can teach a dialogue format. Preferences between replies can then orient the model toward chosen behaviors. Ouyang et al. document one pipeline that uses human feedback. Domain adaptation may also train a small set of additional parameters. Adding a document to a request changes the context at inference time. A parameter update changes the computations on future requests. This distinction determines how to correct information that has become outdated.