Section 1.2 From problem to model

Rules, prediction, generation and decisions

A single system can combine several kinds of computation. A “selected office” field with an agreed meaning can trigger a deterministic rule. A classifier can estimate the office from free text. A generative model can draft a summary for the person reviewing the message. Each output calls for a different test.

A taxonomy by output

TaskOutputPossible criterion
RetrievalRanked list of passagesRelevant passage appears among the top results
ClassificationCategory or scores by categoryPer-class recall and confusion matrix
RegressionNumerical quantityAbsolute error in minutes, dirhams or physical units
GenerationText, image or another structureAccuracy of claims and compliance with the request
DecisionAn action chosen under constraintsError cost, delay and access to recourse

Prediction estimates an unknown quantity. Decision-making applies a policy to that estimate alongside additional information. A 0.70 probability that a ticket needs rapid handling can justify review. Queue capacity and the cost of a delay determine the policy.

Learning associations

A classifier can exploit the association between a phrase and a category without explaining why someone submitted a request. Whether changing that phrase would solve the underlying problem is a causal question. To evaluate an intervention, such as adding guidance to the form, we need a comparison that can distinguish its effect from the enrollment calendar and other changes. A prediction score alone cannot make that distinction.

The same score feeds two different policies, a review queue and automatic transfer.
Figure 1.2. The evaluation of a decision policy must account for its particular consequences.