Hardware for local LLMs: GPUs, Macs and servers
Updated 2026-11-03
Two numbers decide whether hardware can run a large language model well: how much memory it has for the model, and how fast it can read that memory. Compute power matters less than most buyers expect. This guide explains both, then compares the usual options from a single workstation to a company server.
Memory decides what fits
A model’s weights take about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit and a little over 0.5 GB at 4-bit. On top come the KV cache, which grows with the length of the text and the number of requests served at once, and some runtime overhead. A 32B model at 4-bit needs about 18 GB for weights; serving four people with 8,000-token conversations adds about 9 GB of cache. The LLM VRAM calculator does this sum for any model size.
When a model does not fit on one card, a serving engine such as vLLM can split it across several GPUs in the same server.
Bandwidth decides how fast it answers
Every generated token reads all the active weights from memory once. For a single request, the speed ceiling is roughly memory bandwidth divided by the size of the active weights. A card with 1 TB/s running a 20 GB model tops out near 50 tokens per second for one user. Serving engines raise total throughput by batching many requests, so a fast server handles dozens of users at once.
The options
| Option | Memory | Bandwidth (approx.) | Good for |
|---|---|---|---|
| RTX 4090 workstation | 24 GB | 1.0 TB/s | One developer; models up to about 32B at 4-bit |
| RTX 5090 workstation | 32 GB | 1.8 TB/s | One developer or a small team; 32B with long context |
| RTX PRO 6000 Blackwell | 96 GB | 1.8 TB/s | 70B at 4-bit or 8-bit on one card; a small team |
| DGX Spark | 128 GB unified | 0.27 TB/s | Large models for one person or a pilot; slower generation (details) |
| Mac Studio, M3 Ultra | up to 512 GB unified | 0.8 TB/s | Very large models for one person; limited software for serving many users |
| Server with 2 to 8 datacenter GPUs (H100 80 GB, H200 141 GB) | 160 GB to over 1 TB | 3.35 to 4.8 TB/s per GPU | Company use: many users, 70B+ models, production uptime |
Consumer cards (RTX) are the cheapest memory per euro but are not built for 24/7 multi-user service in a server room; check the vendor’s terms and your cooling before putting them in a rack. Datacenter GPUs cost more per gigabyte but come with server chassis, remote management and support.
Matching hardware to the job
One person, experimenting. A workstation with one RTX card, or a Mac with enough unified memory. Ollama or LM Studio make setup easy (comparison).
A pilot for a team. One server with one or two 48 to 96 GB GPUs, or a DGX Spark when traffic is light. Rented cloud GPUs are a good alternative: you measure real load before buying anything.
Production for a company. A GPU server sized from measured load, with vLLM or a similar engine, monitoring, and a plan for failover. Size it in this order:
- Pick the model by testing quality on your own examples.
- Choose the precision (4-bit, 8-bit or 16-bit) that keeps quality acceptable.
- Work out memory for weights plus KV cache at your context length and peak concurrency.
- Check throughput at peak load with a load test, not a single-user demo.
- Compare the monthly cost with API pricing in the LLM cost calculator.
Rent before you buy
GPU rental by the hour lets you run the exact model on the exact GPU type for a week, measure tokens per second under realistic load and see whether quality holds up. Buying hardware after that is a decision based on numbers instead of spec sheets.