Running a model yourself

Open weights vs closed APIs gave the reasons to want open weights. What fits on a GPU did the memory sum that tells you whether a model fits a card. This lesson starts where that one stops: which tool to run it with, what speed you trade away, and the division that says whether it pays. Tool names and defaults here are as of Oct 2026, and every one of them can change, so each claim points at the page to re-read.

1.Three rungs, and the tool that belongs on each

There are three ways to run an open-weight model, and picking the wrong one is the most common self-hosting mistake.

On the first rung, Ollama and llama.cpp run a model on the machine in front of you. On the second, vLLM and SGLang are serving engines. As of Oct 2026, vLLM's docs list continuous batching and prefix caching, and SGLang's docs list prefix caching and compatibility with OpenAI-style APIs. On the third, hosts such as Together and Fireworks sell models through an API, billed per token, with dedicated GPU deployments billed by GPU time (per second at Fireworks, as of Oct 2026). Prices and model lists move, so read each host's own page on the day.

You may still find guides that put TGI, Hugging Face's Text Generation Inference, on the second rung. As of Oct 2026, its docs open with a notice that it is "now in maintenance mode", and they recommend vLLM and SGLang for serving, and llama.cpp or MLX for local use. Its repository has been archived since 21 March 2026. No source I used ranks vLLM against SGLang, so this lesson does not.

The third rung gets less attention than it deserves. It gives you much of what people want from open weights, which is model choice, no surprise retirements and an exit from one vendor, without a GPU on your account. For many teams that is enough, and they skip it because self-hosting sounds like more serious engineering.

The failure to expect is the first rung in front of real users. Ollama's FAQ gives these defaults, as of Oct 2026: one parallel request per loaded model, a 4,096-token context on a card under 24 GiB (the context-length page lists larger defaults for bigger cards), and a queue of 512 requests before it answers 503. Memory also scales with parallel requests times context length. So the model that felt fast to you feels broken to ten people, and the cause is the settings, not the model. You can raise them, but then you are tuning a tool for a job a serving engine was built for.

The rung you use to try a model is rarely the rung you serve it on.

# Try a model on your own machine. No account, no key, no network after the pull.
# Tag as of Oct 2026 - check ollama.com/library for tags that exist when you run this.
ollama pull gemma3:4b
ollama run gemma3:4b "In two sentences, why does VRAM cap which model I can run?"

# The same process serves an OpenAI-shaped API on port 11434.
curl -s http://localhost:11434/v1/models

The last line matters more than it looks. Ollama's docs give http://localhost:11434/v1/ as the base URL, as of Oct 2026, so client code you already wrote for a hosted API points at it by changing one URL. The /v1/models call should return a JSON list that includes the model you pulled, with owned_by set to library by default (as of Oct 2026). A serving engine is the same story: vLLM's quickstart starts its server with vllm serve <model>, on port 8000 unless you change it (as of Oct 2026). Switching models safely builds on exactly this.

2.Throughput against latency, and what quantisation changes

A serving engine gets its efficiency from batching, which means running many requests through the GPU together. More batching gives more total tokens per second across all users, and can make any single answer slower, because your request shares the card. Continuous batching lets requests join and leave a running batch instead of waiting for the whole batch to finish; the vLLM docs list it. You still choose what to tune for: total throughput, or the latency one person feels. You cannot maximise both.

Quantisation, which What fits on a GPU covered for memory, also changes speed, and it does not change it the same way in both regimes. The one study that measured this carefully ran Llama-3.1 models on vLLM 0.6.4 in late 2024. Treat its ratios as indicative, not current: the engine, the kernels and the models have all moved on.

The second effect is on the bill. Fewer bytes means fewer GPUs, and the study's biggest example is the 405B model on 4 GPUs at 4-bit where BF16 needed 16, for roughly 5 to 7 times lower cost per query. Its cost ratios used 2024 rental prices, so do not reuse them. Keep the shape: shrinking a model can cut the GPU count by a whole multiple, and the GPU count is the biggest line in section 3.

Two cautions. The study tested a fixed set of Llama-3.1 and distilled reasoning models and did not test the GGUF files that Ollama and llama.cpp use. And quality can still slip on your cases, so re-run them on the exact file you plan to serve, as Picking a model describes.

3.The idle GPU, and the division that decides

An idle GPU costs exactly what a busy one costs. API pricing falls to zero overnight; a reserved card does not. So the fair comparison is not your peak-hour cost against the API. It is a full month of GPUs, for as many as you need, against a full month of API bill.

Run it as a division: monthly API bill ÷ monthly cost of the GPU boxes you would need, using your cloud's own pricing page on the day. Use at least two boxes, because with one box a single failure takes you down, which is a rule of thumb rather than a measurement.

A worked example, with numbers invented for the sum. Your API bill is 4,000 a month. The smallest box that fits your model rents for 1,500, and you need two, so 3,000. The ratio is 4,000 ÷ 3,000, about 1.3. On paper you save a little. In practice the drivers, upgrades, capacity planning and 3am pages take more time than that saving buys. Now make the bill 20,000 and the ratio is about 6.7. That is worth a serious look, and it gets better the higher and steadier your volume is.

When this division does not apply. The ratio is the wrong test in some cases, because cost is not the reason you are doing it. Self-hosting is clearly right when data legally cannot leave your network, when volume is steady and high, when a fine-tuned model is your actual product, when the model must never change under you, or when the deployment is air-gapped.

When to say no. Self-hosting is probably a hobby when traffic is spiky, volume is modest, nobody on the team can carry the pager, or you need the frontier, because as Open weights vs closed APIs said, the best closed models are not downloadable. In those cases the first or third rung is the honest answer, and the API stays.