Teaching a RAG assistant to say "I don't know"

The first version of our internal assistant answered everything. That was the problem. Ask it about a policy that did not exist and it would produce a fluent, plausible, entirely invented paragraph — and people believed it, because everything else it said had been right.

Confidence is not relevance

A vector search always returns its top k. Cosine similarity of 0.62 to the nearest document does not mean the answer is in that document; it means nothing better exists in the index. Those are very different statements, and the model cannot tell them apart unless you tell it.

We added a floor and a gap check before the model ever sees the context:

record Retrieved(String chunk, double score) {}
 
boolean groundedEnough(List<Retrieved> hits) {
    if (hits.isEmpty()) return false;
    double best = hits.get(0).score();
    if (best < MIN_SCORE) return false;                 // nothing close enough
    if (hits.size() == 1) return true;
    double second = hits.get(1).score();
    return best - second > MIN_MARGIN || second >= MIN_SCORE;
}

When groundedEnough is false, we do not call the model at all. We return a fixed message naming what was searched, and a link to ask a human.

Saying so in the prompt

For the cases that pass the floor, the system prompt has to make refusal an acceptable outcome rather than a failure:

Answer only from the passages provided. If they do not contain the answer, say that the documentation does not cover it and stop. Do not use general knowledge. Every claim must be traceable to a passage.

The instruction that changed behaviour most was the last one. Asking for traceability makes the model check itself, in a way that asking it to "be accurate" never did.

What we measured

We built a set of forty questions we knew the corpus could not answer, alongside a hundred it could.

Correctness went slightly up. Removing the pressure to answer everything also removed a class of confidently-wrong responses to questions that were merely hard.

Trust is asymmetric. One invented policy costs more than fifty helpful answers earn.