Evaluation

Write twenty test questions before you evaluate anything, and write them the way real users ask — half-formed, using internal shorthand, occasionally about things the knowledge base deliberately does not cover. Include at least four questions you know it cannot answer.

Score three things separately: did it find the right material, was the answer right, and did it correctly decline when it should have. A system that scores well on the first two and badly on the third is the dangerous kind, because it fails silently and confidently in exactly the situations where someone is relying on it most.