Comparing AI Providers for Customer Chat

Choosing between the major AI providers looks like a technical comparison and is really a behavioural one. Our guide to choosing an AI model for customer chat frames it around five questions, none of which appear on a public leaderboard, because a model that scores brilliantly on reasoning problems can still be the wrong one to put in front of a customer asking whether you are open on Sunday.

Does it follow instructions it finds inconvenient?

This is the single most important property and the least measured. You will tell your AI never to discuss discounts. A customer will push three times. The question is whether the model holds the line on the third attempt or decides that being helpful outranks the rule.

Test it directly: write one clear prohibition, then spend ten minutes trying to talk the model out of it. Models differ sharply here, and the difference will not show up in casual use — it shows up with your most persistent customer.

Does it admit when it does not know?

Some model families lean toward answering anything; others are more willing to say the material does not cover it. For customer chat, the cautious behaviour is worth a great deal, because a confident invention about a price becomes a commitment. Ask each candidate five questions your documents deliberately do not answer and count the honest refusals — the wider testing method is set out in stopping an AI from inventing answers.

How is it in your customers' languages?

English quality is broadly comparable across the major providers. German and Arabic are not. In German the differences show up in formality — whether the model keeps the formal address consistently through a long thread — and in the fluency of compound business vocabulary. In Arabic they show up in the gap between formal written Arabic and the dialect your customers actually type in, which is where models diverge most.

If a meaningful part of your traffic is in one of these, test in that language and only in that language. A model chosen on English performance can be noticeably worse where it matters.

“

Test the model on your third-most-awkward customer, not your easiest question.

— be digital ai team

Is it fast enough to feel like a person?

In chat, latency is part of quality. A reply that takes fifteen seconds reads as a system thinking; four seconds reads as someone typing. The largest, most capable models are often the slowest, and for routine questions the trade is rarely worth it.

A common pattern is to use a smaller, faster model for straightforward questions and a stronger one where the conversation gets complicated. That works well but adds a moving part, so it is worth doing only once you can see which conversations justify it.

What will it actually cost at your volume?

Headline per-token prices differ by an order of magnitude between the small and large models in the same family, and the practical difference is smaller than that, because most of your tokens are retrieved context rather than the customer's words. Work out the cost of a typical conversation for each candidate rather than comparing rate cards — the arithmetic is in what an AI reply actually costs you.

The things that are not about the model

• Where the provider processes data, and whether a European region is available if you need one.
• Whether the business terms exclude training on your data by default.
• Rate limits on a new account, and how quickly they can be raised before a campaign.
• How often the provider deprecates model versions, since a retired model means an unplanned re-test.

Where a European region is not enough, the alternative is set out in running a local model when data cannot leave. For regulated businesses the first two often decide the question before quality does, which is why they belong in the comparison rather than in a later compliance review — the questions to ask are in where your customer data lives.

Decide with twenty conversations

Take twenty real threads from your inbox, run them through each candidate with identical instructions and documents, and read the results side by side. It takes an afternoon, it answers all five questions at once, and it is worth more than every benchmark published this year — because the only performance that matters is on the conversations you actually get.