Section 1.5 From problem to model
Design an experiment that can fail
A useful experiment can conclude that a model fails to solve the task. In the intake example, define a simple baseline, such as the last category chosen by the submitter followed by human review, and fix an evaluation period. Compare both policies on the same requests and publish the conditions of the test.
What must be held comparable?
A baseline is a fully specified procedure. For weekly forecasting, “use the value from the previous week” is a computable baseline. For routing, “use the menu field when present, otherwise send for review” is also testable. Count the proportion of requests sent for review and the errors among the cases the system handles. An apparent improvement may follow from selecting only the easiest cases.
What to record
The experiment sheet specifies the period, unit of observation, exclusions, data version, code or settings, rubric and most consequential disagreements. A small sample calls for a cautious interpretation. If a metric improves after being used to select a setting, it is a development result. A held-out final set plays a different role, developed in the next chapter.
Synthesis exercise. Reformulate the task for a digital library that wants to extract titles from scanned documents. Propose a baseline, one unit of observation and a criterion that distinguishes exact spelling from preserving document structure. A defensible plan specifies when the scan is available, which pages contain more than one title and what a reviewer can correct.