About 18; min
Running LLMs locally requires an inference engine — the software that loads model weights and generates text. Ollama, vLLM, and llama.cpp are the three most important options, each serving a different level of the stack. Ollama wraps everything in a simple CLI. vLLM provides production-grade serving with the highest throughput. llama.cpp is the low-level runtime that powers most local AI tools. Here’s how they compare.
Quick Comparison
| Feature | Ollama | vLLM | llama.cpp |
|---|---|---|---|
| Primary Use | Easy local LLM running | High-throughput serving | Efficient CPU/GPU inference |
| Target User | Developers, enthusiasts | ML engineers, production | Developers, tool builders |
| Setup | One command install | pip install + config | Build from source or pre-built |
| Model Format | GGUF (quantized) | HF format (full precision/AWQ/GPTQ) | GGUF (quantized) |
| API | OpenAI-compatible | OpenAI-compatible | OpenAI-compatible (server mode) |
| Batching | Basic | Continuous batching (best) | Basic |
| GPU Support | NVIDIA, AMD, Apple Silicon | NVIDIA (CUDA) | NVIDIA, AMD, Apple Silicon, CPU |
| CPU Inference | Yes (via llama.cpp) | No (GPU required) | Yes (primary strength) |
| Concurrent Users | Low-moderate | High (hundreds) | Low |
| License | MIT | Apache 2.0 | MIT |
Ollama Overview
Ollama wraps llama.cpp in a user-friendly package. Install with one command, run
ollama run llama3.1, and chat with a local LLM. The model registry at ollama.com hosts pre-quantized GGUF models that download automatically. The REST API at localhost:11434 follows OpenAI’s format. Ollama handles model management (download, update, delete), GPU detection, memory allocation, and model loading — all automatically. Multi-model support runs different models simultaneously. Custom Modelfiles set system prompts, parameters, and model configurations. Docker support enables server deployment. Ollama’s value is abstraction: it hides the complexity of llama.cpp behind a CLI that anyone can use, while exposing an API that any application can consume. For most developers, Ollama is the right entry point for local LLM work.vLLM Overview
vLLM is a high-throughput LLM serving engine designed for production workloads. Its core innovation is PagedAttention — a memory management technique that handles attention key-value caches like virtual memory pages, achieving 2-4x higher throughput than naive serving approaches. vLLM supports continuous batching (process multiple requests simultaneously without waiting for the longest to finish), speculative decoding (faster generation through draft models), and tensor parallelism (split large models across multiple GPUs). The OpenAI-compatible API server handles hundreds of concurrent requests. vLLM works with Hugging Face model format, supporting full-precision, AWQ, GPTQ, and other quantization methods. For teams serving LLMs to many users simultaneously — chatbots, API services, internal tools at scale — vLLM delivers the highest requests-per-second of any inference engine.
llama.cpp Overview
llama.cpp is the foundational C/C++ library that runs LLMs efficiently on any hardware. Created by Georgi Gerganov, it pioneered GGUF quantization — compressing models to 2-8 bits while maintaining quality, enabling large models to run on consumer hardware. llama.cpp runs on NVIDIA (CUDA), AMD (ROCm), Apple Silicon (Metal), Intel GPUs, and pure CPU. The server mode provides an OpenAI-compatible API. llama.cpp powers Ollama, LM Studio, Jan, and most local AI applications under the hood. Direct use of llama.cpp gives you maximum control over inference parameters, quantization options, and hardware utilization. For tool builders, researchers, and developers who need the lowest-level access to model inference, llama.cpp is the engine that everything else is built on.
Performance Comparison
| Scenario | Ollama | vLLM | llama.cpp |
|---|---|---|---|
| Single user, Llama 8B, RTX 4090 | ~45 t/s | ~50 t/s | ~45 t/s |
| Single user, Llama 8B, M2 Pro | ~35 t/s | N/A (no Metal) | ~35 t/s |
| 10 concurrent users, Llama 70B | ~5 t/s per user | ~15 t/s per user | ~4 t/s per user |
| 100 concurrent users, Llama 70B | Struggles | ~8 t/s per user | Not designed for this |
| CPU only, Llama 8B | ~5 t/s | N/A | ~5 t/s |
| Quantized (Q4), Llama 70B, A100 | ~25 t/s | ~35 t/s (AWQ) | ~25 t/s |
For single-user local use, all three produce similar speeds because Ollama uses llama.cpp internally. The gap appears at scale: vLLM’s continuous batching and PagedAttention serve 10-100 concurrent users with 2-4x better throughput per user. For solo developers, the performance difference is negligible. For production serving, vLLM is in a different category.
When to Use Each
Personal development and prototyping
Ollama. Install in one minute, pull any model, get an API. The abstraction hides complexity you don’t need to manage. Pair with Open WebUI for a visual interface. This covers 90% of local LLM use cases.
Production API serving (10+ concurrent users)
vLLM. Continuous batching, PagedAttention, and tensor parallelism handle concurrent requests efficiently. Deploy behind a load balancer for scaling. If you’re serving an LLM to an application with real users, vLLM is the production standard.
Building tools or custom inference pipelines
llama.cpp directly. When you need control over quantization parameters, sampling strategies, or hardware-specific tuning that Ollama’s abstraction doesn’t expose. Also the right choice when building a new tool that wraps LLM inference (like Ollama itself did).
Apple Silicon Mac
Ollama or llama.cpp. vLLM doesn’t support Metal acceleration. Both Ollama and llama.cpp run efficiently on M1/M2/M3 chips with Metal GPU acceleration. Ollama is the simpler choice; llama.cpp gives more tuning options.
Who Should Pick What
Pick Ollama if:
- You want the simplest path to running LLMs locally
- The model registry and one-command downloads save you time
- An OpenAI-compatible API powers your local AI applications
- Apple Silicon, NVIDIA, or AMD GPU — Ollama handles all three
Pick vLLM if:
- You’re serving LLMs to multiple concurrent users in production
- Maximum throughput per GPU matters for your cost and latency targets
- Tensor parallelism lets you run models across multiple GPUs
- You have NVIDIA GPUs and need enterprise-grade inference performance
Pick llama.cpp if:
- You’re building a tool or library that wraps LLM inference
- Maximum hardware compatibility (CPU, any GPU vendor) is required
- Custom quantization and low-level inference control matter
- You need the lightest possible inference runtime without overhead
Ollama wins for most developers because it provides the right abstraction — simple enough for beginners, powerful enough for production local AI. The one-command setup, model registry, and OpenAI-compatible API cover the vast majority of local LLM use cases. vLLM earns runner-up as the production serving standard — when your LLM serves real users at scale, vLLM’s throughput is 2-4x higher than alternatives. llama.cpp is the essential foundation that powers both Ollama and the broader local AI stack. Choose by your use case: Ollama for development, vLLM for production serving, llama.cpp for building tools.
FAQ
Does Ollama use llama.cpp?
Yes. Ollama uses llama.cpp as its inference backend. Ollama adds model management, API serving, and a curated model registry on top of llama.cpp’s core inference engine.
Can vLLM run on Mac?
Not natively. vLLM requires NVIDIA CUDA GPUs. For Apple Silicon, use Ollama or llama.cpp with Metal acceleration. vLLM support for non-NVIDIA hardware is in development but not production-ready.
Is llama.cpp faster than vLLM?
For single-user inference, they’re comparable. For concurrent users, vLLM is significantly faster due to continuous batching and PagedAttention. llama.cpp wins on hardware compatibility and memory efficiency for quantized models.
Which supports the most models?
Ollama supports any GGUF model (the standard for quantized models). vLLM supports Hugging Face format models. llama.cpp supports GGUF. In practice, all major open-source models are available in formats compatible with all three tools.
