Section 3.1 Generative model architecture
Tokenization and vector representations
A language model receives identifiers. Its tokenizer segments a string and encodes each piece with an identifier from its vocabulary. A boundary can split a word, absorb a space or correspond to bytes. The context consumed by a sentence depends on the tokenizer and the exact form of the text.
Units learned from a corpus
A method based on byte-pair encoding (BPE) starts with primitive symbols. It counts adjacent pairs and repeatedly merges a frequent pair. The merge table is then used to tokenize new text. Variants differ in their initial alphabet, preprocessing and segmentation rules. A common sequence may occupy a single identifier while an uncommon form uses several. Sennrich et al. describe the idea in the context of translation.
IDs, embeddings and contextual representations
The ID is an address. An embedding is the numerical row selected at that address in a learned table. A table with 50,000 tokens and 768 coordinates contains 38.4 million values. Layers then transform the vector using other accessible positions. Two occurrences of the same token start with the same token embedding and can end up with different contextual representations.
Multilingual analysis. Measure tokens per character or word separately for comparable texts in Arabic, French and Latin-script Darija. Record the tokenizer, Unicode normalization and sampling procedure. The ratio describes context use. It does not directly measure answer quality.