The usual process is: open a leaderboard, sort by average score, take the top model that fits the budget, move on. It produces a defensible choice roughly half the time, and the half where it fails, it fails expensively — because switching embedding models means re-embedding everything and rebuilding every index. Here are the questions that predict the outcome better than rank does. Does it work on your text? A benchmark average is a weighted mixture of tasks, most of which are not yours. A model that leads overall can trail badly on the one thing you need — legal clauses, product SKUs, Vietnamese customer support tickets, code. The evaluation that matters takes an afternoon: Collect 50-100 real queries from your logs (or write them, if you have no traffic yet). For each, mark which documents in your corpus should be retrieved. This labelling is the actual work. Embed the corpus with each candidate model, run the queries, measure recall@k for the k you actually feed to the mo...