Glossary

Each term links to the section that explains it and puts it into practice.

Inference
Computing an output from available inputs and a model’s parameters. Section 1.1
Label
Reference value used to train or evaluate a supervised model. Section 1.3
Gradient
Vector of derivatives of the loss with respect to model parameters. Section 1.4
Baseline
A fully specified, typically simple procedure used for comparison. Section 1.5
Self-supervision
Learning with targets derived from the data itself. Section 2.1
Data leakage
Incorporating information that should have been unavailable at the relevant time. Section 2.2
Calibration
Agreement between stated probabilities and observed frequencies. Section 2.3
Precision
Share of flagged cases that really belong to the positive class. Section 2.3
Recall
Share of reference-positive cases that are flagged. Section 2.3
Overfitting
Fit to quirks of the training set at the expense of generalization. Section 2.4
Embedding
Numeric vector associated with an object such as a token or a document. Section 3.1
Token
Unit created by a tokenizer and represented by an ID. Section 3.1
Causal attention
Attention restricted to the current and earlier positions. Section 3.2
Perplexity
Exponential of the mean per-token log loss when using the natural logarithm. Section 3.3
Logit
Score before conversion to a probability. Section 3.4
Context window
Maximum number of tokens that can be used together for a request. Section 3.4
Diffusion
Family of generative methods that learn to reverse progressive noising. Section 3.5
Transfer
Ability to apply a principle to a problem that differs from the original exercise. Section 4.1
RAG
Generation that receives passages retrieved from a collection at request time. Section 4.4
Provenance
Information tracing the origin, version and handling of data. Section 5.2
Coverage
Share of requests that the system chooses to answer. Section 5.4