Section 5.4 Reliability and capstone

Trade off reliability, cost and latency

A larger model or longer context may improve certain answers and make the service more expensive. A useful comparison evaluates several dimensions on the same requests. Define response time from the reader’s perspective, the passage budget and the cost of requests a person must revisit.

Measure observable tradeoffs

The median time describes a typical request under the observed protocol. The 95th percentile is a threshold below which roughly 95% of times fall, depending on the estimator. A mean does not reveal long requests that can make a system difficult to use. Recording retrieval, input processing and generation separately helps locate the delay. A service may return a passage promptly and prepare a synthesis once it has enough evidence.

Two latency distributions have similar medians and a longer tail in the second configuration. The median and 95th percentile are marked.
Figure 5.4. Constructed times. Similar typical performance can coexist with very different response times in the tail.

Account for abstention

A policy that handles 60 out of 100 requests and answers 54 correctly has 90% accuracy on its answers and 60% coverage. Another handles 90 and gets 72 right, yielding 80% accuracy on its answers and 90% coverage. The two create different workloads for follow-up. Compare consequential errors, coverage and review capacity. In production, also record retries, incomplete replies and source failures.