Section 2.5 Learning and evaluation
Deep networks and learned representations
A network builds representations by composing transformations. A dense layer computes z = Wx + b, then applies an activation such as ReLU(z) = max(0, z) to each coordinate. Stacking affine operations alone still yields an affine transformation. Nonlinearities allow richer decision boundaries.
From the forward pass to the update
The forward pass computes predictions and a loss. Backpropagation applies the chain rule to the computation graph to obtain gradients for the parameters. The optimizer uses those gradients to decide on a parameter update. This distinction gives three places to investigate a badly specified loss, nearly zero gradients or an excessive learning rate.
Transfer a representation
A pretrained encoder can produce vectors for a small supervised classifier. Freezing the encoder reduces the number of trainable parameters. Partial fine-tuning may adapt the representation to a domain, but increases cost and the risk of overfitting the available cases. Compare a word- or character-based baseline and an encoder on the same test to determine whether any gain justifies the cost. In vision, convolutions exploit spatial locality. In language, transformers organize interactions between positions, the subject of Chapter 3.
Synthesis question. If the encoder raises the overall score but makes rare cases harder to identify, review per-class errors and the permitted decision policies. The choice depends on the output and its cost, as in the opening section of this book.