Seven questions that shorten an AI procurement from months to weeks

The bottleneck in AI procurement is rarely the vendor. It is that the buying organization has no agreed basis for saying yes, so the evaluation expands until someone senior decides on instinct. A short, fixed set of questions applied consistently does more for decision speed than a longer requirements document, because it makes the answer defensible rather than merely arrived at.

The questions

What does the system do when it is uncertain? A capability with no abstention path will answer confidently in exactly the cases where it should not, and the cost of that lands on whoever is downstream.

What was it evaluated against, and how does that compare to our data? Vendor benchmarks are run on data that flatters the system. The relevant question is performance on documents, language, and edge cases resembling yours — and a vendor unwilling to be evaluated on a sample of your real data is telling you something.

Who reviews the output, at what rate, and is that capacity staffed? Human-in-the-loop appears in most proposals and is resourced in few. A review step nobody has time to perform is an unfunded control, and it will be quietly dropped within a quarter.

What happens to our data — is it retained, and is it used for training? This needs a contractual answer, not a reassurance in a meeting. It is also the question most likely to be discovered late, during an audit, by which point the integration is load-bearing.

How is a wrong result detected after the fact? Systems degrade silently as inputs drift. If the only detection mechanism is a customer complaint, the organization is running an unmonitored process and calling it automation.

What would we do if this capability were withdrawn, repriced, or moved behind a different licence? The most commonly skipped question and the most expensive to answer late.

What does one unit of this cost to run, and what drives that number up? Pilot pricing rarely survives production volume, and the variable that scales the bill is often not the one in the proposal — document length, retry behaviour, retrieval breadth, or a per-seat floor that applies whether or not the seat is used. Ask which of those the vendor meters on, then model the bill at ten times the pilot volume rather than at the pilot.

Model the bill at ten times the pilot volume. Pilot pricing rarely survives production.

Why these seven

Each maps to a failure that reaches production regularly: overconfident output with no abstention path, evaluation on data unlike the buyer’s, review capacity that was never staffed, retention terms found during an audit, silent degradation, dependency on a capability that moved behind a different commercial model, and unit economics that only worked at pilot scale.

None of them require machine-learning expertise to ask, and all of them require it to answer well — which is the correct division of labour between a buyer and a vendor.

How this compresses the timeline

Evaluations stretch when the criteria are discovered during the process. Fixing the questions in advance turns an open-ended assessment into a bounded one: seven answers, a sample evaluation on your own data, and a decision. The vendors who answer well are visibly better, and the ones who cannot are disqualified early rather than after a lengthy pilot — which is where most of the wasted cost in AI procurement sits, since a pilot consumes integration effort from the buying organization whether or not it leads anywhere.

It also produces a durable artefact. The answers become the baseline you monitor against once the thing is live, which means the procurement work is not thrown away at signature — it becomes the first version of the oversight process the system will need anyway.