Seven questions to ask an AI vendor
A short list that separates teams who have shipped from teams who have demoed. Useful whether or not the vendor you are evaluating is us.
Buying AI work is harder than buying most software, because the demo is the easiest part to fake and the hardest part to evaluate. These questions tend to sort it out quickly.
1. What does this cost to run per month at our volume?
A good answer is a number with the assumptions attached. A vague answer means nobody has run it at scale.
2. What happens when the model is wrong?
Listen for a specific mechanism — a confidence threshold, a review queue, a fallback. "We use a really good model" is not an answer.
3. Can we see a trace of a single request?
Teams who have operated a system in production have this to hand, because they have spent weeks staring at it. Teams who have not will offer a diagram instead.
4. How is this evaluated, and can we see the set?
Ask for the evaluation set and how it was built. If evaluation is "we tried it and it looked good," you are the evaluation.
5. Who owns the model, the prompts and the data?
Get it in writing, including what happens on the day you leave. This is easier to agree before the engagement than after.
6. What is the handover?
Runbooks, dashboards, an on-call story and someone on your side who understands the system. A project that only works while the vendor is in the room has not finished.
7. What would you refuse to build?
The most revealing question on the list. A vendor who will build anything you describe has no opinion about what works, and you are paying for the opinion.
Got a problem that looks like one of these?
Tell us what you are trying to build or automate. An engineer reads every enquiry personally.