You have a RAG system, no users yet, and a prompt you keep changing. Every change feels like an improvement and you have no way to tell. Writing a hundred test questions by hand takes two days and you will not do it. So you ask a model to generate them. This works, and it is worth doing, and it has one failure mode that invalidates the whole exercise if you do not design around it. The circularity problem If you generate questions by showing a model your documents, you get questions your documents answer well, phrased the way your documents phrase things. Your retrieval then scores beautifully — on a test set constructed from the same assumptions as the system under test. Real users ask about things your documents cover badly, in words your documents never use, with false premises baked in. A synthetic eval set built naively measures internal consistency, not usefulness. It will pass while your users fail. Everything below is about breaking that circle. Seed from real artefacts,...