Section 1.3 From problem to model

Measuring and labeling data

A column can be easy to export and hard to interpret. If the target is the last office that touched a request, the model may reproduce internal workflows instead of representing the person’s need. If the target is the first office equipped to respond, humans will need to annotate it. In either case, documenting what the target means tells us more than knowing the column name.

The label as a measurement

Write a short annotation guide with borderline examples and a procedure for disagreement. Have two readers independently annotate the same batch before adjudication. Raw agreement is a first observation to read alongside the class distribution. When nine out of ten requests fall into one category, high overall agreement may leave ambiguity in the other categories unresolved. Keeping a record of why reviewers disagree helps improve the guide or redesign the task.

Why provenance matters

A minimal record associates each example with a group identifier, timestamp, collection channel and annotation guide version. A form submission has a different structure from a repeatedly forwarded email. A query mixing French and Darija might express one issue in two ways. Comparing results by channel and language pattern helps separate collection problems from model limitations. Chapter 5 returns to precautions during collection and to uncertainty intervals.

Four provenance fields accompany the same example row, including time period, channel, group and annotation guide version.
Figure 1.3. Provenance metadata support quality checks and splitting. Assess separately whether each field is available at inference.

Analysis prompt. A redesigned form includes explicit categories. A model trained before the redesign suddenly sees much shorter messages. The distribution of inputs has changed. The useful measurement now comes from new-form data, with a target whose definition either remains stable or has been explicitly revised.