Section 5.5 Reliability and capstone
Capstone · A verifiable documentation assistant
Build a prototype that looks up explanations in a bounded collection of technical documents. One topic is the behavior of lists and functions in Python. The objective is to show when an assistant provides justified help and under which conditions it needs more evidence or should abstain.
Build a reproducible baseline
- Index at least twelve passages from public documents. Record their version, ID, section heading and address.
- Write at least fifteen questions, including five the corpus cannot answer and several French–English pairs. Freeze a final set before tuning the system.
- Implement a lexical baseline. For each answerable question, keep the expected evidence and the result of a local test when one is available.
- Measure hit@k on answerable questions. Report coverage, evidence support and correct abstentions, with denominators.
Present the results
Compare the baseline with one variant, such as chunking that preserves headings or an assistant that turns retrieved passages into hints. Freeze the variant before the final test. Show a success, a case with missing evidence, a case with misused evidence and a correct abstention. Include a result from an unaided transfer task. The report includes nonpersonal data, reproducible examples and a conclusion bounded by the experiment. The Chapter 2 audit script provides reusable metric definitions.