Section 2.2 Learning and evaluation

Split the data to reflect future use

Randomly assigning rows to sets is reasonable when rows are sufficiently independent and future use follows the same collection process. A request history often contains several messages from the same case. If messages from a case appear in both training and test, the model may recognize a familiar thread. To predict next month’s requests, a time split is often more faithful, with groups kept from crossing the chosen boundaries.

Three responsibilities

Training fits parameters and any learned transformations, including the vocabulary or the mean used to normalize a variable. Validation helps select the representation, hyperparameters and action threshold. Testing measures the chosen solution after those decisions are fixed. With limited data, grouped cross-validation can stabilize development choices. Keep an independent test set if you need a final estimate.

A timeline places training before validation and test. Three messages from one thread stay in one block.
Figure 2.2. For prospective use, a time split recreates the information available when a decision is made.

Two types of leakage

Target leakage introduces a field derived from the future, such as the office that closes a request. Protocol leakage lets test information influence the choice of method, for example by fitting TF-IDF on the full corpus before splitting. Even if the vocabulary contains no labels, its document frequencies now reflect documents meant to be unseen. A pipeline fits the learned transformation and classifier together within each split.

Documented case. The scikit-learn documentation shows how headers, signatures and quoted text in 20 Newsgroups can indirectly reveal a post’s topic. The experiment is useful because it exposes a verifiable shortcut that can also appear in service archives.