When local AI deployment is the right answer

Most businesses should use a cloud AI API. It’s cheaper, faster to implement, and requires zero ops work. We say this to clients often.

There is, however, a narrower set of situations where local deployment is the right answer. This post is about recognizing which situation you’re in.

The default answer is “use an API”

If you’re a general small business (a restaurant, a trades company, an e-commerce shop, an early-stage SaaS), the cloud AI providers (OpenAI, Anthropic, Google) are the correct choice. You pay per request, there’s no infrastructure to manage, and the models are excellent.

We build cloud-API-backed features for most of our clients. It usually takes days of engineering, not months.

When that answer changes

There are a few specific cases where cloud APIs stop being viable:

Regulated industries with data-processor restrictions.

• Healthcare (PHIPA in Ontario, HIPAA in the US). Patient-identifiable information typically cannot be sent to a third-party processor without a signed agreement (a BAA under HIPAA; a written services agreement plus Canadian data residency under PHIPA) and a specific legal review. The major cloud AI providers (OpenAI’s API, Anthropic, Google’s Vertex AI, AWS Bedrock, Azure OpenAI) will sign a BAA and offer the data controls on enterprise plans, but not on their default consumer products.
• Legal (attorney-client privilege). Sending privileged communications to a third party can, depending on the jurisdiction and the firm’s position on the question, be treated as a waiver. Firms are risk-averse here for good reasons.
• Finance (PCI-DSS, SOC 2, banking regulations). Transaction data, PII, and account information often have controls that preclude a third-party AI API by default.

Government or defense contracts. Data classification and residency rules make this one straightforward.

Contractual restrictions from your customers. We’ve seen B2B SaaS products whose enterprise customers prohibited any third-party subprocessor. If one of your clients has that clause, your AI stack needs to respect it.

What “local deployment” actually involves

Local deployment means running a model on infrastructure you control, either on-prem or in a private cloud tenant (a dedicated VPC in AWS, Azure, or GCP).

In practice, the setup is:

1. An open-weight model (Llama 3.x or 4, Qwen3, Gemma 3, Mistral, or a domain-specific fine-tune).
2. A GPU or small cluster sized for your expected traffic.
3. An inference server (vLLM, SGLang, or Ollama) that exposes an HTTP endpoint.
4. A thin API layer that your application calls instead of api.openai.com.
5. Logging, monitoring, and access controls that meet your compliance requirements.

The work isn’t magical, but it isn’t trivial either. For a single-GPU 7B-model deployment, a reasonable estimate is two to four weeks of engineering, plus ongoing ops time measured in hours per month.

What it doesn’t do

Local deployment solves the data-processor question. It does not, by itself, solve any of these:

• Data residency within a specific country (you still need to pick your region).
• Encryption at rest and in transit (you still need to configure it).
• Access controls (still your responsibility).
• Audit logging (still required for compliance).
• Model quality (a smaller local model may be meaningfully worse at some tasks than GPT-5 or Claude).

If someone sells you local deployment as a full compliance solution, push back.

The honest tradeoff

Local deployment is more expensive to build and maintain than a cloud API call. The reason to do it is not cost; it’s control. You own the inference pipeline, the data never leaves your environment, and you can answer “where does this data go?” with a one-line answer.

For regulated industries, that answer is often worth the extra engineering. For most other businesses, it isn’t.

How we approach it

When a client asks about local deployment, we work through a short checklist:

1. Is there a specific regulatory or contractual reason this can’t be a cloud API? If not, we usually recommend the cloud API.
2. What’s the use case? A narrow, well-defined task (classification, extraction, summarization) is much easier to serve locally than an open-ended assistant.
3. What’s the realistic volume? A handful of requests per hour is different from a few requests per second, and the hardware sizing is very different.
4. Who will operate this after launch? If there’s no one, we either run it on a retainer or push back on the decision.

Only after that do we scope the actual build.

Update, August 24, 2026. This is the oldest post on this site and the framing above has held up better than the specifics. The decision rule is unchanged: local deployment buys control, not savings, and the reason to do it is a regulatory or contractual constraint rather than a cost model. What has moved since early 2025 is the arithmetic and the shortlist of models, both of which are covered properly in newer pieces rather than patched into this one. For the money, see the real total cost of running your own LLM in 2026, which works through hardware, power and staffing and shows where the cloud break-even actually lands. For the workload shapes where self-hosting genuinely wins, see when local LLMs beat cloud APIs. And if the constraint driving you here is Ontario health privacy law specifically, the PHIPA build checklist is the more direct answer.

Update, August 25, 2026: one thing this post understated. The architecture section above lists an inference server such as vLLM, SGLang or Ollama as though standing it up were the end of the job. It is not, and the gap has become measurable. Ollama ships with no authentication at all, and a botnet documented in July queries Shodan by name for exposed Ollama, Langflow, n8n, Open WebUI, ComfyUI and Gradio instances specifically to harvest the cloud credentials sitting next to them. Over the same period CISA added Langflow to its actively exploited catalog five times, plus n8n, Ray and MLflow. If you take the local route for the privacy reasons argued above and then leave the serving layer reachable, you have traded a vendor’s security team for nobody’s. We wrote the operational side up separately in the piece on self-hosted AI tools and the KEV catalog, which includes the ports to check and the questions to ask. Read it before you deploy, not after.

If you’re trying to figure out whether local AI applies to your situation, we can walk through it with you.