What it actually costs to run AI on your own hardware
A plain-English breakdown of the hardware, GPU, memory, and cloud costs involved in running AI inference — for operators who want a number, not a whitepaper.
Most businesses asking about “running their own AI” don’t actually want to train models. They want to run inference — send text or an image in, get output out — on their own infrastructure, usually for privacy or cost reasons.
Here’s what that costs in 2026, without the marketing.
Three separate cost centers
Running AI on your own infrastructure has three distinct line items:
- Inference — running a trained model to produce output.
- Training or fine-tuning — teaching a model on your data.
- Storage and retrieval — the database that feeds relevant context to the model.
Most clients we work with only need the first. If someone is proposing you train your own foundation model, get a second opinion.
Inference: the only number most people need
For a model to run, the whole thing has to fit in GPU memory (VRAM). That’s what sets the hardware floor.
| Model size | VRAM needed | Typical GPU | Monthly cost (cloud) |
|---|---|---|---|
| 7–8B params | ~16 GB | NVIDIA L4, A10, RTX 4090 | $400–$900 |
| 13–34B params | 32–80 GB | A100 (40/80 GB) | $1,500–$3,500 |
| 70B+ params | 140 GB+ | 2× A100 or H100 | $5,000–$15,000 |
Quantization (compressing the model weights from 16-bit to 4-bit) can cut VRAM requirements by roughly 4×, which is how people run a 70B model on a single A100.
On-prem pricing is different. A single-GPU workstation suitable for a 7B model runs roughly $3,000–$6,000 to buy outright. The economics only make sense if the box is going to run every business day for two or more years.
When a cloud API is cheaper
The honest answer is: most of the time, for most businesses.
If you’re sending fewer than about 10 million tokens a month to a model, a cloud API (OpenAI, Anthropic, Google) will cost you under $150. That’s cheaper than a GPU instance sitting idle overnight and on weekends. The math only flips when you have either:
- High, steady volume. 40+ million tokens per month, roughly 24/7.
- A privacy constraint. Patient data, attorney-client privileged communications, financial records, or anything with a compliance framework that forbids third-party processors.
If neither of those applies, the cloud APIs are both cheaper and easier to operate. We tell clients this regularly.
The memory constraint
The short version: GPU memory is the bottleneck for most inference, not compute. Everything you hear about H100s being hard to get is a symptom of this.
High Bandwidth Memory (HBM) — the specialized memory inside datacenter GPUs — is effectively sold out for 2026 and into 2027. The knock-on effect is higher prices for normal DRAM too, because memory manufacturers make more margin on HBM and have shifted capacity accordingly. If you’re buying hardware this year, budget 20–40% more than you did in 2024.
Storage and retrieval
If your model needs to answer questions about your own content — product catalogs, support docs, contracts — you need a retrieval layer. In practice that’s a vector database (Pinecone, Qdrant, Supabase pgvector, Weaviate) holding embeddings of your content.
Cost ranges:
- Small (under 100k documents): $25–$100/month on a managed service, or free on Supabase pgvector if you already have Postgres.
- Medium (under 10M documents): $200–$1,000/month.
- Large: custom pricing, and usually a dedicated team to run it.
For most of our clients, the answer is pgvector on their existing Postgres. It’s fine until you’re very, very large.
Three realistic scenarios
Small business with a support backlog. Use a cloud API. $50–$200/month. No GPU, no ops work.
Clinic, law firm, or finance team with a privacy requirement. Run a 7–13B open-weight model on a single-GPU cloud instance in your VPC, or on an on-prem workstation. $800–$2,500/month all-in, plus setup.
High-volume SaaS product with AI as a core feature. Start on cloud APIs until you hit roughly $3,000/month in API spend, then move the highest-volume, most-repetitive workloads onto your own GPUs. The hybrid setup is almost always the cheapest answer.
Common mistakes
A short list of things we’ve watched clients spend money on that they didn’t need:
- Buying GPUs before validating the use case. Start on an API. Measure real usage. Then decide.
- Picking the largest available model by default. A fine-tuned 7B often beats an off-the-shelf 70B on narrow tasks.
- Ignoring idle cost. Cloud GPUs bill by the hour whether you use them or not. Batch your work.
- Treating AI as a one-time project. Once it’s in production, someone has to update the model, monitor outputs, and fix regressions. That’s ongoing engineering time.
If you’re trying to size infrastructure for a specific use case, we can help run the numbers.