Running a Local Model When Data Cannot Leave

Somewhere in most AI conversations, someone asks whether the model can run on our own servers. Sometimes that is a serious requirement and sometimes it is discomfort looking for a technical shape. Our guide to choosing an AI model for customer chat treats it as a real option with real costs, worth taking seriously and worth talking yourself out of if the reason does not hold.

When it is genuinely necessary

Three situations make self-hosting the right call. A regulator or professional body that prohibits sending client information to third-party processors — some legal and clinical contexts qualify. A contract with an enterprise customer that specifies where their data may be processed. Or an internal policy, written before AI existed, that would take longer to change than to comply with.

Notice what is not on the list: general unease about cloud providers. If the concern is that a model provider might train on your customers' messages, the fix is a business agreement that forbids it — the terms to look for are described in data processing agreements and subprocessors.

The middle options people skip

Between a public API and a machine in your office there are two intermediate answers that solve most cases with far less effort.

• A hosted model pinned to a specific region, so processing stays in the EU or another jurisdiction you have committed to.
• An enterprise cloud deployment where the model runs inside your own cloud account, with your own access controls and logging, but you are not maintaining hardware.

For the great majority of businesses one of these satisfies both the regulator and the customer, without anyone becoming responsible for keeping a GPU alive. Check them first — the residency questions are covered in where your customer data lives.

“

"Can it run locally?" is usually the right instinct expressed as the wrong requirement. Ask what the rule actually says first.

— be digital ai team

What it actually takes

Running an open-weight model that is good enough for customer conversations means a machine with a serious GPU, not a spare office computer. You will also need someone who can keep it running: patching, monitoring, restarting it when it stops answering at seven on a Friday evening, and re-testing whenever you upgrade to a newer model.

The hardware cost is one-off and knowable. The maintenance cost is ongoing and usually underestimated, because it lands on whoever is technical enough to fix it — which in a small business is often the person least able to spare the time.

Where the quality lands

Open-weight models have improved enormously and are entirely capable of answering routine customer questions from a knowledge base. Where they still lag is in the harder behaviours: holding a rule under pressure, staying consistent across a long conversation, and handling non-English dialects with the fluency of the largest hosted models. If your customers write in Gulf dialect Arabic, test that specifically and do not assume.

Test locally-run candidates against the same twenty conversations you would use for any other model, with the method in testing an AI before you let it talk to customers. A model that is slightly worse but keeps you compliant is a good trade; one that is noticeably worse for a reason you cannot articulate is not.

The economics are not the reason

Self-hosting rarely saves money at small-business volumes. Hosted APIs are cheap per conversation and you pay only for what you use; a GPU costs the same whether it answers a thousand messages a month or none. The break-even is far higher than most teams expect, and it moves further away every time hosted prices drop.

Choose local for control and compliance, never for cost. If the business case rests on savings, work through what an AI reply actually costs you with real numbers before committing.

A practical compromise

Some businesses split it: a local model handles anything touching sensitive records, while general enquiries — opening hours, prices, availability — go to a hosted model. It doubles the setup and it is genuinely how several regulated teams operate, because it puts the expensive constraint only where the rule actually applies rather than across every conversation you have. Whichever route you take, write down the reason. In two years someone will ask why the setup is unusual, and "our professional body requires it" is a much better answer than institutional memory.