How to choose an AI development company: 10 questions that separate builders from deck-sellers

Since we sell custom AI builds ourselves, a buying guide from us deserves your suspicion. So here is the deal: everything below is a question you can ask any vendor, including us, and the good answers are described concretely enough that you can score the responses yourself. If this list disqualifies us on your project, it did its job.

Why this decision is hard right now

Two years ago, building anything with AI required rare skills. Today a working demo takes a weekend, which means demos no longer carry information. The real difference between vendors is everything that happens after the demo: whether output quality is measured or vibes-based, whether failures are handled or discovered by your customers, and whether you own the result or rent it. None of that is visible in a sales meeting unless you ask for it directly.

The 10 questions

1. "What have you shipped that real users depend on?"

Not "what have you built" — what runs in production, today, with users who would notice if it broke. Listen for a specific story: the problem, the users, what went wrong, what they changed. A portfolio of proofs-of-concept that never went live is the single most common failure signature in this market.

2. "How do you measure whether the AI is good enough?"

The answer should contain the word evals, or something that means it: a test set of real cases, a quality bar agreed with you before the build, and regression runs so next month's change doesn't quietly break last month's behaviour. "We iterate until you're happy" sounds collaborative and means nothing is measured.

3. "What happens when it's wrong?"

Every AI system produces wrong output sometimes. The vendor's job is a designed failure path: guardrails on what the system may do, confidence thresholds, fallbacks, and a human-in-the-loop wherever a mistake costs real money or trust. A vendor who only talks about the happy path has not shipped much.

4. "Who owns the code, the prompts, and the data at the end?"

The right answer is: you. Code in your repository, infrastructure in your cloud accounts, prompts and eval sets included in the handover. If the deliverable only runs on the vendor's closed platform, you did not buy software — you rented it, and the rent is renegotiated at their convenience.

5. "What does the first two weeks look like?"

Good answer: access to your data and systems, a thin working slice of the real thing, and a scope document with a definition of done. Bad answer: a discovery phase, a workshop series, and a strategy deck. Discovery that costs six figures and produces a PDF is a consulting business model wearing an engineering costume — we wrote about the difference in AI consulting vs AI engineering.

6. "Will our data be used to train anything shared?"

You want explicit answers on three things: whether your data trains models used for other customers (it should not, without written consent), where the data is processed and stored, and which third-party AI services sit in the pipeline. If your business runs under DPDPA, GDPR, or sector rules, ask how the architecture respects that — and expect specifics, not "we take security seriously".

7. "Which models do you use, and why?"

There is no single right answer, but there is a right shape: a reasoned, current view that matches models to tasks and costs, holds no religious loyalty to one lab, and expects to swap components as the market moves. A vendor who cannot explain why they chose a model will not notice when it becomes the wrong choice.

8. "What will this cost to run after launch?"

The build is a one-time cost; model calls, infrastructure, and human review time are forever. A serious vendor gives you a run-cost estimate per month or per task alongside the build quote. We broke down how those numbers behave in the economics of AI agents, and what drives build price in how much custom AI costs.

9. "How do we exit?"

Ask what handover looks like if you part ways in a year: documentation, deployment runbooks, eval sets, and a codebase your own engineers (or a different vendor) can pick up. Vendors build switching costs by default. Make the exit path part of the contract, not a hope.

10. "What part of our problem will AI not fix?"

The most predictive question on the list, because it tests honesty rather than competence. Every real project has a part where the answer is process, data cleanup, or plain software with no model in it. A vendor who claims AI solves all of it is selling you the word, not the work.

Red flags that end the conversation

• Accuracy guarantees. "99% accurate, guaranteed" is not how probabilistic systems work. Honest vendors talk about eval scores on your data, and how they improve them.
• A quote before questions. Anyone who prices your build before asking about your data, your systems, and your users is pricing a template, and you will pay the difference later.
• Open-ended hourly billing. With no fixed scope and no definition of done, the incentive is the meter, not the finish line. Insist on fixed scope, fixed price.
• Prototype-only portfolio. Demos and hackathon wins are not production. See question 1.
• Buzzword density. If the proposal says transformational more often than it says your industry's actual nouns, the writing is the product.

Buying software vs buying a build

One distinction before you start interviewing: if an off-the-shelf product already fits your workflow, buying it beats building — faster, cheaper, maintained by someone else. Our build vs buy framework walks through that decision, and our vendor evaluation framework covers the buying-software case. This post is for when the answer came back "build".

Where we stand

For transparency, here is how Xwits custom builds answer the questions above: fixed scope and fixed quote after we understand the problem, a working slice early instead of a discovery deck, evals agreed before we write the system, human-in-the-loop wherever stakes are real, code and infrastructure in your accounts, and a four-to-eight-week delivery target. We are early and we are small — that is exactly why we compete on the answers to these ten questions rather than on logo walls.

What this means for you

• Interview at least two vendors and ask all ten questions. The comparison is where the signal is.
• Weight questions 1, 2, and 10 highest: shipped production work, real evals, and honesty about limits predict the rest.
• Get IP ownership, run-cost, and exit terms into the contract, not the conversation.
• Walk away from guarantees, hourly meters, and pre-emptive quotes without hesitation.

Want to run your project past us — including these ten questions? Book a 30-minute call. If we are not the right fit, we will say so and tell you what to look for instead.