Section 3.3 Generative model architecture
Pretraining, adaptation and post-training
For tokenized text t₁, …, tₙ, an autoregressive objective asks each position to predict the next token. During training, the observed continuation supplies the target. Negative log-likelihood strongly penalizes targets assigned very small probabilities. A probability of 0.1 costs approximately 2.303 nats, compared with 0.223 at 0.8.
What does perplexity measure?
Perplexity is the exponential of average loss per token when the loss uses the natural logarithm. A model that always spreads its probability uniformly across 10 possibilities has perplexity 10 on text whose targets belong to that vocabulary. Compare perplexities with the same corpus, segmentation and evaluation protocol. This metric measures prediction of observed text. A task such as providing accurate citations needs its own evaluation.
Change the model or change its request
Supervised fine-tuning on demonstrations can teach a dialogue format. Preferences between replies can then orient the model toward chosen behaviors. Ouyang et al. document one pipeline that uses human feedback. Domain adaptation may also train a small set of additional parameters. Adding a document to a request changes the context at inference time. A parameter update changes the computations on future requests. This distinction determines how to correct information that has become outdated.