Section 2.4 Learning and evaluation
Regularization, overfitting and change
The model that best minimizes training loss can encode incidental details. In older requests, a standardized signature might identify an office with striking accuracy. If the signature format changes, that strategy collapses. Fit to the past and generalization are two separate measurements.
Diagnose with learning curves
A learning curve can show training and validation loss at several stages. If both are high, first check the data, representation and optimizer. If training loss keeps falling while validation loss rises, one option is to stop at a stage selected on validation. Other options include reducing capacity, penalizing large weights or improving the data. Reducing complexity does not necessarily address a different input distribution.
Investigate a production regression
A change can affect the distribution of inputs, the frequency of targets or the relationship between an input and its target. More requests at the start of term might alter category frequencies while preserving the definition of urgency. A new procedure could change the correct category for previously identical messages. Reviewing annotation practices and current reference data helps distinguish these mechanisms.
Diagnosis. Keep a frozen set for regression checks and collect a recent batch to measure the current domain. Report both scores with their time periods. A pooled average can hide an improvement on the old request stream and a decline on the new one.