How to Run an LLM Locally

Running a model on your own machine has got dramatically easier. It is now a single command for most people, and the interesting questions have moved from how to which model, at what quantization, and is this actually worth it.

Why you would

The data never leaves. This is the reason that survives every other argument. If you cannot send customer records, patient data, or unreleased source code to a third-party API, local inference is not a preference, it is the only option.

Cost at volume. Per-token pricing is excellent value until you are processing millions of tokens of routine work — bulk classification, extraction, summarisation over a large corpus. At that point hardware you already own starts to win.

No rate limits, no outages, no deprecations. A local model does not get retired out from under you, which matters if you have tuned prompts against its specific behaviour.

Experimentation. Trying twelve models costs you disk space rather than twelve billing relationships.

Why you would not

Be honest about this before investing a weekend.

The quality gap is real. Open-weight models have improved enormously and the best of them are genuinely good. They are still, generally, behind the frontier hosted models on hard reasoning, long-context work, and reliable instruction-following at the edges. For most production work where quality determines outcome, a hosted frontier model remains the better answer.

Your hardware is the ceiling. You cannot prompt your way past insufficient memory.

You have become an operator. Updates, model management, serving, monitoring. Fine on a laptop; a project on a server.

The sensible default for most teams is hybrid: local for the high-volume, low-stakes, privacy-sensitive work; hosted for the hard reasoning. Which model handles which is exactly the question in Choose the Right Model for a Task — define the hardest case the task must handle, then test on your own data.

The tools

Ollama (ollama/ollama, ~178,600 stars) is where almost everyone should start. Install it, run one command, and you have a model running with an HTTP API on localhost. It handles downloading, quantization selection, and memory management for you. The API is close enough to the common shape that most client libraries work with a changed base URL.

LM Studio is the desktop GUI equivalent — a model browser, a chat window, and a local server, with no terminal required. Better for exploring what runs well on your machine; it also ships a CLI (lmstudio-ai/lms).

llama.cpp (ggml-org/llama.cpp, ~124,000 stars) is the C/C++ inference engine underneath a great deal of this ecosystem, including Ollama. Go here directly when you want control — specific quantization formats, specific flags, embedding it in an application. Note the repository moved from ggerganov/llama.cpp to the ggml-org organisation, so older links and instructions point at the wrong place.

vLLM (vllm-project/vllm, ~89,100 stars) is a different category: a high-throughput serving engine for GPUs, built for concurrency rather than single-user chat. This is what you deploy when a service needs to serve many simultaneous requests, not what you install on a laptop.

Star counts retrieved from the GitHub API on 15 August 2026.

Quantization, which is the concept that actually matters

Model weights are trained at 16-bit precision. Quantization stores them at lower precision — 8-bit, 5-bit, 4-bit — which shrinks memory requirements roughly proportionally, at some cost to quality.

This is the lever that decides what you can run. The rough arithmetic:

Memory ≈ (parameters × bytes per parameter) + overhead for the context.

A model at 16-bit needs about 2 bytes per parameter. At 8-bit, about 1. At 4-bit, about 0.5. So the same model quantized to 4 bits needs roughly a quarter of the memory of the full-precision version. That is frequently the difference between running on the machine you have and not running at all.

The quality cost is not linear. Dropping from 16-bit to 8-bit is usually close to imperceptible. Going to 4-bit is generally a good trade and the most common choice. Below that, degradation becomes noticeable and gets worse quickly.

Two rules of thumb worth internalising:

A larger model at lower precision usually beats a smaller model at higher precision, at equal memory. If you can fit either a big model at 4-bit or a small one at 8-bit, try the big one first.

Add headroom for context. The weights are not the only thing in memory — the KV cache grows with how much context you feed it, and a long prompt on a model that just fits will fail or spill to disk. Leave room.

Hardware, briefly

Apple Silicon punches above its weight because unified memory is shared between CPU and GPU, so a Mac with a lot of RAM can run models that would need an expensive discrete GPU otherwise. This is why so much local-LLM activity happens on Macs.

On PC, VRAM is the binding constraint — not the GPU's speed, its memory. A card with more VRAM and fewer cores will run a bigger model than the reverse, and running a bigger model is usually what you want.

CPU-only works and is slow. Genuinely fine for batch work you leave running; frustrating for interactive use.

The practical approach is to start with a small model, confirm the whole pipeline works, then step up in size until you hit the wall. You will find your machine's limit faster by bisecting than by researching.

Making it actually useful

Once it is serving on localhost, most tooling treats it as any other endpoint. That means the rest of the discipline applies unchanged and matters more, not less, because you are working with a smaller model:

• Smaller models are more sensitive to prompt quality. Structure earns more here than it does with a frontier model — the anatomy of a great prompt.
• Few-shot examples do disproportionate work on smaller models. If output format is drifting, three examples usually fix it — few-shot vs zero-shot.
• Do not assume the sampling defaults are right. Temperature and sampling parameters vary more visibly on smaller models.
• Tool calling support is uneven across open-weight models and much less reliable than on frontier models. Test it specifically before building on it — what is function calling.
• Measure before you conclude. The question is not whether a local model is as good in general, but whether it is good enough for your task — which is an eval, not an opinion. How to evaluate and test your prompts.

Deliberately absent here: a table of specific models and their memory requirements. Open-weight releases move monthly and any such list is misleading within a season. Ollama's and LM Studio's own model catalogues are current; this page will not be.

The short version

Install Ollama, pull a small model, confirm it works, then scale up until your memory runs out. Understand quantization, because it decides everything. Expect a quality gap and measure whether it matters for your task rather than assuming either way. Use local for volume and privacy, hosted for hard problems, and stop treating it as a loyalty test.

More in the AI development pack, and how to prompt open-source and challenger models covers getting good output once it is running.

Sources

Repository metadata retrieved from the GitHub API on 15 August 2026:

• ollama/ollama — ~178,600 stars; local model running and serving
• ggml-org/llama.cpp — ~124,000 stars; LLM inference in C/C++, the engine underlying much of the ecosystem, relocated from ggerganov/llama.cpp
• vllm-project/vllm — ~89,100 stars; a high-throughput and memory-efficient inference and serving engine for LLMs
• lmstudio-ai/lms — the LM Studio CLI