Section 5.1 Reliability and capstone

Audit performance by use case and language

An overall result weights cases in the same proportions as the sample. If French queries dominate a dataset, its average can conceal errors on Arabic or Darija queries. Language categories themselves need definitions. A query may switch languages mid-sentence, contain a proper name or use several spellings. The labeling guide should distinguish language from difficulty and request type.

Interpret the gaps

In a constructed example, a system answers 90 out of 100 questions correctly in set A and 12 out of 20 in set B. The pooled rate is 102/120, or 85%. The set-level rates are 90% and 60%. A 95% Wilson interval for B is approximately [39%, 78%] if observations are independent. Twenty paraphrases contributed by the same person may be correlated. Treating them as independent can understate uncertainty. These results can guide further collection and decide which failures merit analysis.

Points at 90% and 60% with Wilson intervals based on 100 and 20 observations. A reference line marks the pooled result at 85%.
Figure 5.1. Constructed data. Group size affects both estimate precision and the weight of the group in the pooled result.

Find the upstream cause

Compare matched tasks and inspect segmentation, transcription, retrieval and scoring errors separately. If an assistant misses a French passage when queried in Arabic but replies correctly when the passage is supplied, retrieval failed in that case. If the passage was present and explained incorrectly, inspect generation and the scoring rubric. You can then test a targeted intervention on similar cases and a broader sample.