Nobody is lying to you in an AI demo, which is what makes it dangerous.
The person driving has run that sequence two hundred times. They know which phrasings work, which questions land in the middle of the agent's competence, and which one produces the awkward pause. They aren't hiding the failures so much as they've stopped visiting them, the way you stop taking the route with the bad junction.
The demo is a road the vendor has driven two hundred times.
Bring your own twenty
Before you take a single vendor call, pull twenty real cases out of your own business. Emails, call recordings, tickets, forms, whatever the agent is supposed to handle.
- Twelve ordinary ones. The bread and butter. If it fumbles these, nothing else matters.
- Five hard ones. The customer who asks three questions in one message. The one who's angry. The one whose request is reasonable and outside your normal service.
- Three your own staff get wrong. The genuinely ambiguous ones. You're not checking whether the agent gets these right, you're checking whether it knows it's out of its depth.
Don't send them in advance. Run them live on the call, in the vendor's system, with you reading them out. A vendor who won't do that has told you something useful and you can end the meeting twenty minutes early.
Score it in front of them
Three columns, filled in as you go: handled, escalated, wrong.
Escalated isn't a failure. An agent that hands off the three hard cases is behaving correctly, and say so out loud, because it changes what the vendor demonstrates next.
The column that matters is wrong, and specifically confidently wrong. One confidently wrong answer in twenty is a 5% failure rate on work that reaches your customers with your name on it. Ask what they'd do about that specific case. The answer tells you whether you're buying a product or a demo.
Five questions that end a bad call fast
1. What does it refuse to do?
A vendor with a real answer has a list and reaches for it easily. A vendor who says "it can handle pretty much anything" has never defined the boundary, which means you'll be the one who finds it.
2. Show me a case it fails.
Every honest system has some. Watching somebody walk through a failure and what they decided to do about it is the most informative two minutes of any vendor call.
3. What happens when the model underneath you changes?
These systems sit on top of models that get updated. Ask how they find out, what they re-test, and what happened the last time. "We test everything carefully" is not a process. The vocabulary for judging that answer is worth twenty minutes before the call.
4. If we leave, what do we take?
A year of configuration — your rules, your escalation logic, your tuned prompts — is a real asset. Can you export it, and in what form? Most SMBs never ask, and lock-in ends up being the thing they pay for without ever having priced it.
5. Who gets woken up when it breaks at 6pm on a Friday?
A name and a channel, or it's a ticket queue. Both are fine answers. Only one of them matches what the salesperson implied.
The pilot trap
You'll be offered a free 30-day pilot and it will feel like the low-risk option. Usually it isn't, for two reasons.
Most pilots measure nothing. Thirty days pass, everyone forms an impression, and the decision gets made on vibes with a month of setup already sunk. If you run one, write down the pass mark before it starts: the specific number, on the specific metric, against a baseline you took first.
A pilot is also the best service you will ever receive. You have their attention now. Ask directly what support looks like in month six at your contract value, and pay attention to the gap between the two answers.
Reference calls, done properly
Every vendor will hand you a delighted customer. Three questions get past the script.
- How long did onboarding actually take, versus what you were told? The honest answer is usually double, which is survivable, but it should be a number you knew rather than a surprise.
- What did you have to change about how you work? Every real deployment forces some change. A reference who can't name one hasn't deployed it deeply.
- Can I talk to somebody who downgraded or left? Almost nobody asks this. The reaction to the question is informative even when the answer is no.
Terms an SMB forgets to price
- Your data. Where it goes, how long it's kept, whether it's used to improve anything outside your account. Get it in writing regardless of what you're told on the call.
- Renewal pricing. Ask what the increase was for existing customers last year. The introductory rate is not the price.
- Volume behavior. What happens in your busiest month: does it degrade, queue, or bill you differently?
- Notice period. Ninety days on an annual contract means you decide about year two in month nine.
What a good vendor sounds like
Specific and slightly boring. They volunteer limits before you find them. They'll tell you about a case where their product is the wrong choice. They give you a number when you ask for one, including when the number isn't flattering.
We're on the vendor side of this table, so take the whole piece as an invitation to run it on us. Ask us the five questions. Most of our answers are written down already in our definition of done, and if we can't defend them on a live call with your twenty cases in front of us, we haven't earned the work.