Section 5.5 Reliability and capstone

Capstone · A verifiable documentation assistant

Build a prototype that looks up explanations in a bounded collection of technical documents. One topic is the behavior of lists and functions in Python. The objective is to show when an assistant provides justified help and under which conditions it needs more evidence or should abstain.

Build a reproducible baseline

  1. Index at least twelve passages from public documents. Record their version, ID, section heading and address.
  2. Write at least fifteen questions, including five the corpus cannot answer and several French–English pairs. Freeze a final set before tuning the system.
  3. Implement a lexical baseline. For each answerable question, keep the expected evidence and the result of a local test when one is available.
  4. Measure hit@k on answerable questions. Report coverage, evidence support and correct abstentions, with denominators.
Four deliverables follow one another, corpus, questions, baseline and error report.
Figure 5.5. The prototype is a comparison instrument. The report traces observed failures to different parts of the system.

Present the results

Compare the baseline with one variant, such as chunking that preserves headings or an assistant that turns retrieved passages into hints. Freeze the variant before the final test. Show a success, a case with missing evidence, a case with misused evidence and a correct abstention. Include a result from an unaided transfer task. The report includes nonpersonal data, reproducible examples and a conclusion bounded by the experiment. The Chapter 2 audit script provides reusable metric definitions.