Free Interactive Tool
RAG Similarity Threshold Calibrator
Too low a threshold and noisy chunks slip through. Too high and you over-filter real matches. Drag the slider to find the cutoff that maximises faithfulness on real ShopBot queries — then read the chapter to understand the math.
Interactive Tool · Chapter 6
Similarity Threshold Explorer
20 real queries from run_20260722_085310.json. Drag the threshold and watch what passes vs. gets blocked.
20
pass threshold
0
blocked
60%
accuracy (passing)
0.75
0.550.650.750.850.95
0.500.98
Hover over a dot to see query details.
5 wrong answers still pass your threshold of 0.75. At the default 0.75, the system answers but makes mistakes. The threshold controls what enters the pipeline, not whether the answer is right.
CorrectPartialWrongBlocked
Book 1 · Chapter 6 — Threshold Noise
Why the default threshold is almost always wrong
Chapter 6 of The AI Engineer explains the failure mode this tool exposes — why similarity scores cluster between 0.75 and 0.91, how to pick a calibration dataset, and the iterative tuning loop ShopBot used to raise faithfulness from 0.71 to 0.91.
Read Chapter 6 — Free →