• Technical • long

llama.cpp: A Complete Technical Reference — History, Internals, Usage, and Cloud Deployment

The de facto local-LLM inference engine: GGML/GGUF internals, quantization (Q4_K_M, imatrix, IQ types), backends, llama-server OpenAI-compatible API, conversion and quantization workflow, and Azure/AWS deployment — a technical reference.

#llama.cpp#gguf#ggml#quantization#inference#llm#local-llm#llama-server#deployment

GGML · GGUF · Quantization · llama-server · Conversion Workflow · Azure & AWS Deployment

Part of the Univrs ML reference series. Companion pieces: Post-Training Techniques and Training Platforms (how models get trained), Azure GPU and CPU Cost Reference (what the hardware costs), and Machine-Learned Surrogates for Feed Formulation (where ML surrogates and amortized optimization actually help).

TL;DR

  • llama.cpp is the de facto standard engine for running LLMs locally: a minimal-dependency C/C++ inference library created by Georgi Gerganov on March 10, 2023 (ten days after Meta’s LLaMA weights leaked), built on his GGML tensor library, and now the hidden engine inside Ollama, LM Studio, GPT4All, KoboldCpp, and most GGUF-based tools. On February 20, 2026, Gerganov’s ggml.ai team joined Hugging Face while keeping the project MIT-licensed and community-governed.
  • Its architecture is quantization-first and portability-first: the GGUF single-file format bundles weights, tokenizer, and metadata; k-quants and importance-matrix (imatrix) quantization compress models to 2–8 bits; and a pluggable backend layer (CPU SIMD, CUDA, Metal, Vulkan, SYCL, ROCm/HIP, RPC) runs the same computation graph on almost any hardware.
  • Choose llama.cpp for single-user, edge, Apple Silicon, CPU, or mixed CPU/GPU inference and maximum portability; choose vLLM/SGLang for high-concurrency GPU serving. On cloud, run it on CPU (AWS c7g Graviton, Azure Fsv2/Dv5) for cost-efficient light loads or GPU (AWS g5/g6, Azure NC A100 v4) for latency, using the official ghcr.io/ggml-org/llama.cpp Docker images.

Key Findings

  1. Origin. Gerganov started GGML in late September 2022 (inspired by Fabrice Bellard’s LibNC), shipped whisper.cpp first, then hacked the first llama.cpp “in an evening” after the LLaMA leak. The original README’s stated goal was “to run the model using 4-bit quantization on a MacBook.”
  2. Format evolution. The original GGML file format (and its GGMF/GGJT iterations) lacked extensible metadata and broke backward compatibility whenever features were added. GGUF (GPT-Generated Unified Format), released August 21, 2023, replaced it with typed key-value metadata, embedded tokenizer, and single-file self-description. GGUF is now the dominant local-LLM format on Hugging Face; the old .bin GGML format is deprecated and unsupported.
  3. Ecosystem. Ollama, LM Studio, GPT4All, KoboldCpp, text-generation-webui, and llama-cpp-python all depend on llama.cpp or GGML. ggml.ai was founded in 2023 with backing from Nat Friedman and Daniel Gross; it was acquired by Hugging Face in February 2026.
  4. Quantization. Legacy quants (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0), k-quants (Q2_K–Q6_K with S/M/L mixes), and i-quants (IQ1–IQ4, codebook-based, need an imatrix). Q4_K_M is the recommended default sweet spot; imatrix quantization cuts perplexity loss materially, especially below Q4.
  5. Backends & distribution. CPU (AVX/AVX2/AVX512/AMX/NEON), CUDA, Metal, Vulkan, SYCL, HIP/ROCm, MUSA, CANN, plus an RPC backend for splitting a model across machines and --split-mode/-ts for multi-GPU.
  6. Server. llama-server exposes an OpenAI-compatible API with chat/completions, embeddings, multimodal (libmtmd/vision+audio), GBNF grammar-constrained JSON, and function/tool calling via --jinja.
  7. Governance/status. Continuous b#### build-tagged releases (around b10400+ by August 2026), 124K GitHub stars as of August 14, 2026, CMake build system (old Makefile/LLAMA_CUBLAS deprecated), CUDA flag -DGGML_CUDA=ON.

Details

1. History & Context

The GGML foundation (2022). In late September 2022, Georgi Gerganov — a Bulgarian software engineer — began work on GGML (Georgi Gerganov Machine Learning), a lightweight tensor-algebra library written in C with a focus on strict memory management, multi-threading, and zero runtime allocations. Its design was inspired by Fabrice Bellard’s LibNC. The ggml.ai project describes its aims as “AI at the edge,” “no third-party dependencies,” and “zero memory allocations during runtime.” Before llama.cpp, Gerganov used GGML to build whisper.cpp, a C/C++ port of OpenAI’s Whisper speech-recognition model.

The LLaMA moment (March 2023). Meta released the LLaMA model weights to researchers in February 2023; within days the weights leaked publicly via a torrent. Meta’s original implementation depended on PyTorch + FairScale and required CUDA/NVIDIA hardware, putting it out of reach for most developers. Gerganov ported the inference code to raw C++ over a single weekend, enabling a 7B model to run on a MacBook CPU. llama.cpp was first released on March 10, 2023 — ten days after the leak. The original README stated: “The main goal is to run the model using 4-bit quantization on a MacBook. […] This was hacked in an evening — I have no idea if it works correctly.” Writing the next day (March 11, 2023), Simon Willison called the moment decisive: “OK, I’m calling it: Large language models are having their Stable Diffusion moment right now… This all changed yesterday, thanks to the combination of Facebook’s LLaMA model and llama.cpp by Georgi Gerganov.”

Evolution & the GGUF format. The original GGML file format stored weights efficiently but carried insufficient metadata: it could not reliably identify which architecture a model used, and adding a new setting could silently break older loaders. It went through iterations (GGML → GGMF → GGJT). On August 21, 2023, the project introduced GGUF (GPT-Generated Unified Format), a successor based on GGJT but redesigned to be extensible and unambiguous. GGUF stores model info as typed key-value metadata that loaders read selectively (unknown keys are ignored), so new architectures add parameters without breaking existing files. It is a single-file deployment format bundling weights, tokenizer, and architecture metadata. GGUF is now the dominant format for distributing quantized local LLMs on Hugging Face; the old .ggml/.bin format is obsolete and no longer supported by llama.cpp.

Spinoff projects and the ggml org. llama.cpp is co-developed alongside GGML under the ggml-org GitHub organization, which also hosts whisper.cpp and the standalone ggml library. Gerganov’s optimization of Apple Silicon’s Unified Memory Architecture (via ARM NEON and Metal) made MacBooks one of the best local-inference platforms.

Downstream ecosystem. llama.cpp is “the de facto standard as the core of almost all local inference tools”:

  • Ollama — the most popular local-LLM wrapper, built on llama.cpp; founded 2021 by Jeffrey Morgan and Michael Chiang (YC W21), launched publicly 2023 as “Docker for LLMs.” It has been criticized for obscuring its llama.cpp dependency.
  • LM Studio — a polished desktop GUI using llama.cpp (and MLX on Mac); it added speculative decoding in v0.3.10.
  • llama-cpp-python (by Andrei Betlen) — Python bindings offering a low-level API, a high-level API, an OpenAI-compatible web server, and LangChain/LlamaIndex compatibility.
  • GPT4All, KoboldCpp, text-generation-webui, Open WebUI, LlamaBarn — all build on llama.cpp/GGUF.

Governance, funding, and status. ggml.ai was founded by Gerganov in 2023 to sustain llama.cpp development, originally backed by Nat Friedman and Daniel Gross. On February 20, 2026, Hugging Face announced that Gerganov and the GGML team were joining the organization. Under the arrangement, the team became full-time Hugging Face employees while retaining 100% of their time on llama.cpp, full technical autonomy, and full ownership of the open-source trajectory; Hugging Face provides long-term sustainable resources. The projects stay MIT-licensed and community-driven. The stated goals: near “single-click” deployment from Hugging Face’s 1-million-model hub to local inference, tighter integration between the transformers library (source of truth for model definitions) and llama.cpp (optimized inference), and faster quantized-model support after new releases. Two HF employees, ngxson (Xuan-Son Nguyen, “Son”) and allozaur (Aleksander Grygier, “Alek”), were already core llama.cpp contributors — as HF CEO Clem Delangue put it in the official announcement, “we even have awesome core contributors to llama.cpp like Son and Alek in the team already” — so the deal formalized existing collaboration. The project surpassed 100,000 GitHub stars in March 2026 — a milestone that took PyTorch about seven years and TensorFlow closer to eight, but which llama.cpp reached in under three. As of August 14, 2026 the repo stands at 124K stars (with over 20,000 forks) and contributions from 700–900+ developers.

2. Functional & Architectural Approach

Design philosophy. llama.cpp is deliberately minimal: “LLM inference in C/C++” with no heavyweight runtime dependencies (no PyTorch, no CUDA developer toolkit in the inference path). It compiles to small native binaries that run on Linux, macOS, Windows, Raspberry Pi, Android, and in-browser via WebGPU. It is CPU-first with optional GPU acceleration and quantization-first, letting 7B+ models run in 4–8 GB of RAM.

GGML tensor library fundamentals. The core data structure is ggml_tensor, an n-dimensional array (up to 4 dims) storing shape (ne[]), byte strides (nb[]), a data type, and computational provenance. Row-major, non-contiguous tensors are supported, enabling zero-copy view operations (reshape, permute, transpose). Computation uses a lazy-evaluation model: the user defines operations that build a directed acyclic graph (ggml_cgraph) of nodes; nothing computes until ggml_graph_compute() / ggml_backend_graph_compute() is called, which traverses nodes in topological order. This separation of definition from execution allows memory planning and graph optimization ahead of time. The backend scheduler (ggml_backend_sched) distributes ops across available hardware using a multi-pass assignment algorithm; if an op is unsupported on a GPU backend, it automatically falls back to CPU and inserts tensor-copy operations at backend boundaries, so exotic ops don’t block GPU offload of the rest of the graph. All backends implement the uniform ggml_backend_i interface, and backends can be compiled in statically or loaded dynamically as plugins (GGML_BACKEND_DL).

GGUF file format deep dive. GGUF is a binary container designed for fast loading and self-description. It stores: a magic/version header, typed key-value metadata (architecture name, hyperparameters, chat template, tokenizer vocab/merges/special tokens), and the tensor data with per-tensor type info. Because the tokenizer is embedded, no external tokenizer.json/config.json is required at runtime. Extensibility (readers ignore unknown keys) is precisely what let llama.cpp add hundreds of architectures without redesigning the loader. GGUF is versioned; the current major version has been stable while metadata keys expand.

Quantization techniques, step by step. Quantization reduces the precision of weights (e.g., FP16 → 4-bit integers), shrinking memory ~4× and speeding memory-bandwidth-bound inference, at some accuracy cost. llama.cpp implements quant types as custom GGML tensor types operating on fixed-size blocks/super-blocks, leaving sensitive tensors (norms, embeddings, output) at higher precision:

  • Legacy formats: Q4_0 (symmetric 4-bit), Q4_1 (adds per-block offset/zero-point), Q5_0/Q5_1, Q8_0. Generally superseded by k-quants at similar sizes.
  • k-quants (mixed-precision per-super-block): Q2_K, Q3_K_S/M/L, Q4_K_S/M, Q5_K_S/M, Q6_K, Q8_K. The _S/_M/_L suffixes denote small/medium/large mixes (how many tensors get more bits).
  • i-quants (IQ): IQ1_S/M, IQ2_XXS/XS/S/M, IQ3_XXS/XS/S/M, IQ4_XS, IQ4_NL — codebook-based, best at very low bit widths, but they degrade badly without an importance matrix.
  • Other: BF16, F16, ternary TQ1_0/TQ2_0, and MXFP4 (used by gpt-oss).

Importance-matrix (imatrix) quantization uses calibration data to compute per-weight importance statistics, so the quantizer preserves the weights that matter most. It typically reduces perplexity loss by 10–30% versus naive quantization and is essentially mandatory for IQ types and recommended below Q5_K_M. Recommendation by size: small models (1–3B) → Q5_K_M/Q6_K to preserve quality; 7–14B → Q4_K_M (the ~75%-size-reduction sweet spot) or Q5_K_M; 70B → Q3_K_M/Q4_K_M; 100B+ → Q2_K/IQ with imatrix to fit memory. Measure quality loss with perplexity (ppl) and KL-divergence.

Backend architecture. GGML abstracts compute behind pluggable backends, each enabled at build time by a CMake flag (combinable, e.g. -DGGML_CUDA=ON -DGGML_VULKAN=ON):

  • CPU: AVX/AVX2/AVX512/AMX (x86), NEON/SVE (ARM), optional BLAS. Supports both x86_64 and aarch64.
  • CUDA (-DGGML_CUDA=ON): NVIDIA via cuBLAS + custom kernels (mmq/mmvq etc.), Flash Attention, multi-GPU. CUDA 12 and CUDA 13 image variants exist.
  • Metal (-DGGML_METAL=ON): Apple Silicon, native path exploiting unified memory.
  • Vulkan (-DGGML_VULKAN=ON): cross-platform GPU via SPIR-V shaders; by 2026 it “quietly closed most of the token-generation gap” and is the default on ARM64 Linux (e.g., Qualcomm X Elite). The universal fallback that runs on almost any GPU.
  • SYCL (-DGGML_SYCL=ON): Intel Arc GPUs via oneAPI/DNNL/XMX.
  • HIP/ROCm (-DGGML_HIP=ON, superseding the old -DGGML_HIPBLAS): AMD Radeon/Instinct; often ports CUDA kernels.
  • Others: MUSA (Moore Threads), CANN (Huawei Ascend), OpenCL (Adreno), WebGPU, Hexagon DSP, IBM zDNN, OpenVINO, VirtGPU, ZenDNN, and RPC for distributed inference.

Model loading & inference pipeline. Load GGUF → build the architecture-specific compute graph → tokenize input (tokenizer embedded in GGUF) → prefill the prompt (batched, compute-efficient) → autoregressive decode. The KV cache stores per-layer key/value tensors so each new token attends to prior context without recomputation; it can be quantized (--cache-type-k, --cache-type-v, e.g. q8_0) to save memory, and --flash-attn reduces its footprint and speeds attention. Context management is set by -c/--ctx-size. Batching (-b, --parallel, --cont-batching) enables concurrent request handling. Sampling strategies include temperature, top-k, top-p (nucleus), min-p, typical, tail-free, Mirostat v1/v2, repetition/frequency/presence penalties, and GBNF grammar-constrained sampling.

Grammar-constrained output (GBNF). GBNF (GGML Backus-Naur Form) is an extended BNF with regex-like features that restricts the sampler at each step to only tokens that keep the output valid — guaranteeing e.g. syntactically valid JSON. llama.cpp converts a subset of JSON Schema to GBNF automatically; via the server /chat/completions endpoint you pass response_format with json_object or json_schema. Note the schema only constrains output — it is not injected into the prompt — and grammar sampling adds roughly 8–15% decode overhead; there is a known issue where grammar sampling can hang on very long contexts (>10k tokens) with some dense (non-MoE) models. (An optional LLGuidance backend, -DLLAMA_LLGUIDANCE=ON, adds faster CFG/JSON-Schema constrained decoding but requires the Rust toolchain.)

Server architecture (llama-server). A lightweight C++ HTTP server with a built-in web UI and an OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models), plus native endpoints (/completion, /embedding, /props, Prometheus /metrics). It supports embeddings (--embedding), multimodal input via libmtmd (image_url as URL/base64/local path), function/tool calling (--jinja, optionally with a --chat-template-file override), reasoning/thinking controls, and continuous batching. Many env vars mirror flags (e.g., LLAMA_ARG_MODEL, LLAMA_ARG_N_GPU_LAYERS).

Multimodal (libmtmd). Vision/audio support was reworked in 2025 into libmtmd (replacing llava.cpp), unifying model-specific CLIs into llama-mtmd-cli and later bringing vision to the server (PR merged May 2025, led by ngxson). It uses a separate multimodal projector file (mmproj) that encodes media into embeddings interleaved with text tokens. Supported: LLaVA 1.5/1.6, Gemma 3, Qwen2-VL/Qwen2.5-VL/Qwen3-VL, InternVL 2.5/3, MiniCPM-V, Pixtral, DeepSeek-OCR (vision); Ultravox, Qwen2-Audio, Voxtral (audio); Qwen2.5-Omni (both). Generate mmproj via convert_hf_to_gguf.py --mmproj.

Multi-GPU & distributed inference. Two multi-GPU modes: --split-mode layer (default — layers assigned serially to GPUs; VRAM scales but only one GPU computes at a time) and --split-mode row (weight matrices sharded across GPUs so all compute in parallel; benefits most from NVLink; NCCL build helps). --tensor-split/-ts controls the proportion per GPU. The RPC backend (-DGGML_RPC=ON) serializes each tensor op over TCP to rpc-server processes on other machines, letting you split one model across a cluster (e.g., a Mac Studio + a Linux/NVIDIA box). Important caveat: RPC lets you fit a bigger model, but does not make tokens faster than a single machine that could already hold the model.

Speculative decoding. A small draft model proposes several tokens that the large target model verifies in one parallel pass; accepted tokens are free, output is identical to the target alone. Enabled via -md/--model-draft (draft and target must share a compatible vocabulary). In the 2026 CLI rework, the tuning flags were renamed under a --spec- prefix (--spec-draft-n-max/--spec-draft-n-min, --spec-type), and new self-speculative strategies were added (n-gram cache, n-gram map, plus architecture-native MTP/Eagle drafters in models like Qwen3.x and Gemma). Speedups of ~1.5–3× occur when the acceptance rate is high (70%+); on a tight GPU a draft model can make you slower.

3. Layered Practical Usage Guide

Layer 1 — Basics on macOS (Apple Silicon)

Install via Homebrew (simplest — gives you llama-cli, llama-server, etc.):

brew install llama.cpp
llama-cli --version

Or build from source with Metal (for the latest features):

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j
export PATH="$PWD/build/bin:$PATH"

(Metal is on by default on Apple Silicon; -DGGML_METAL=ON is explicit. Binaries land in build/bin/.)

Run inference — download a GGUF directly from Hugging Face and offload all layers to the GPU:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -ngl 99 -p "Explain quantization in one paragraph."

Or point at a local file:

llama-cli -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  -p "Write a haiku about tensors." \
  -n 256 -c 8192 -ngl 99 --temp 0.7 --top-p 0.95

Key CLI flags: -m model path; -hf download from HF; -n/--predict max tokens to generate; -c/--ctx-size context window; -ngl/--n-gpu-layers layers to offload to GPU (use 99 to offload all); -b/--batch-size; -t/--threads; --temp, --top-k, --top-p, --min-p, --repeat-penalty sampling; -cnv conversation mode; --flash-attn on. On an M3/M4 expect ~60–120 tok/s on a 7–8B Q4 model.

Layer 2 — Running a local server

llama-server -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 16384 -ngl 99 --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --cont-batching --parallel 4

Open http://localhost:8080 for the web UI. Call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Hello!"}]}'

Use it from any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-noauth")
resp = client.chat.completions.create(
    model="local",
    messages=[{"role":"user","content":"Summarize llama.cpp in one sentence."}])
print(resp.choices[0].message.content)

Python integration with llama-cpp-python:

# CPU wheel
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
# Build with Metal
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python
# Build with CUDA
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

from llama_cpp import Llama
llm = Llama(model_path="./models/qwen2.5-7b-instruct-q4_k_m.gguf",
            n_gpu_layers=-1, n_ctx=8192)
print(llm.create_chat_completion(
    messages=[{"role":"user","content":"Hi"}])["choices"][0]["message"]["content"])

It also ships its own OpenAI-compatible server: python3 -m llama_cpp.server --model model.gguf --host 0.0.0.0 --port 8000.

Layer 3 — Model conversion and quantization workflow

# 0. Install Python deps (from the llama.cpp repo root)
python3 -m pip install -r requirements.txt

# 1. Convert HF safetensors -> GGUF (FP16 by default)
python3 convert_hf_to_gguf.py ./models/mymodel/
# or with explicit type/output:
python3 convert_hf_to_gguf.py Qwen/Qwen3-8B --outtype bf16 --outfile Qwen3-8B-BF16.gguf
#   --outtype options: {f32, f16, bf16, q8_0, tq1_0, tq2_0, auto}
#   other flags: --remote (convert straight from HF hub), --mmproj (multimodal projector)

# 2. (Recommended) Build an importance matrix from calibration text
./build/bin/llama-imatrix -m Qwen3-8B-BF16.gguf -f calibration-data.txt -o imatrix.gguf -ngl 99
#   default output is now imatrix.gguf (GGUF); legacy: --output-format dat -o imatrix.dat
#   --chunk N sets the starting chunk

# 3. Quantize (Q4_K_M, using the imatrix)
./build/bin/llama-quantize --imatrix imatrix.gguf \
  Qwen3-8B-BF16.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M

The minimal official three-step sequence (no imatrix) is:

python3 -m pip install -r requirements.txt
python3 convert_hf_to_gguf.py ./models/mymodel/
./llama-quantize ./models/mymodel/ggml-model-f16.gguf ./models/mymodel/ggml-model-Q4_K_M.gguf Q4_K_M

Useful llama-quantize options: --pure (uniform type, disables k-quant mixtures), --leave-output-tensor, --output-tensor-type, --token-embedding-type, --tensor-type (regex per-tensor targeting), --include-weights/--exclude-weights, --prune-layers, --keep-split, --allow-requantize, --dry-run. For multimodal models, convert and quantize the vision/audio projector separately. Note the flag is --chunk (singular), and the default imatrix output is now GGUF (imatrix.gguf); older guides showing imatrix.dat reflect the still-supported legacy --output-format dat.

Layer 4 — Building simple open-source applications

Minimal chatbot — just point any OpenAI SDK at llama-server (see Layer 2).

LangChain:

from langchain_community.llms import LlamaCpp
llm = LlamaCpp(model_path="./models/model.gguf", n_gpu_layers=-1, n_ctx=8192, temperature=0.7)

Or against the server via langchain-openai’s ChatOpenAI(base_url="http://localhost:8080/v1", api_key="x").

LlamaIndex:

pip install llama-index-llms-llama-cpp

from llama_index.llms.llama_cpp import LlamaCPP
llm = LlamaCPP(model_path="./models/model.gguf", model_kwargs={"n_gpu_layers": -1})

Simple RAG using llama.cpp for both embeddings and generation. Run one server with an embedding model and one with a chat model:

llama-server -m nomic-embed-text-v1.5.Q8_0.gguf --embedding --port 8081
llama-server -m qwen2.5-7b-instruct-q4_k_m.gguf --port 8080

Embed documents via POST /v1/embeddings to 8081, store vectors in a store (FAISS/Chroma), retrieve top-k at query time, and pass the retrieved context to the chat model on 8080. LlamaIndex/LangChain can orchestrate both through their llama.cpp integrations.

Layer 5 — Advanced

Grammar-constrained JSON:

curl http://localhost:8080/v1/chat/completions -d '{
 "messages":[{"role":"user","content":"Extract name and age as JSON."}],
 "response_format":{"type":"json_schema","json_schema":{"schema":{
   "type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},
   "required":["name","age"]}}}}'

Or with the CLI: llama-cli -m model.gguf --grammar-file grammars/json.gbnf -p "...".

Function/tool calling (requires --jinja):

llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M

Then POST an OpenAI-style tools array to /v1/chat/completions. Native tool-call formats: Llama 3.1/3.2/3.3, Functionary v3.1/3.2, Hermes 2/3, Qwen 2.5 (+Coder), Mistral Nemo; a generic handler is the fallback. Avoid extreme KV quantization (e.g. -ctk q4_0) as it degrades tool-calling.

Speculative decoding:

llama-server -m qwen2.5-coder-14b-instruct-q4_k_m.gguf \
  -md qwen2.5-coder-0.5b-instruct-q8_0.gguf \
  --spec-type draft-simple -ngl 99

Multi-model serving: run multiple llama-server instances on different ports (or use the router mode), and use a reverse proxy to route by model name.

Batching for throughput: add --cont-batching --parallel N and size -b/-c appropriately; benchmark with llama-batched-bench and llama-bench.

4. Cloud Deployment — Azure

VM SKU selection.

  • CPU-only / light loads: Fsv2 (compute-optimized) or Dv5/Dsv5 (general-purpose) — cheapest, fine for small quantized models at low concurrency.
  • GPU inference: NCasT4_v3 (NVIDIA T4 16 GB; Standard_NC4as_T4_v3 on-demand pricing starts at $0.1050/hr in East US 2 for the smallest size — good for 7B Q4 models) and NC A100 v4 (NVIDIA A100 80 GB PCIe; Standard_NC24ads_A100_v4 at $3.673/hr on-demand or $0.67877/hr spot). The newest line is NCads H100 v5 (~$6.98/hr for a single H100 NVL, Standard_NC40ads_H100_v5). Microsoft notes it is deploying net-new capacity only for NCads_H100_v5. Every new Azure subscription starts with zero N-series vCPU quota, so file a quota-increase request first.

Build on an Azure Linux VM with CUDA.

# Install NVIDIA driver + CUDA toolkit (or use the NVIDIA GPU Driver Extension)
sudo apt-get update && sudo apt-get install -y build-essential cmake git
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)
./build/bin/llama-server -m model.gguf -ngl 99 --host 0.0.0.0 --port 8080

Note the old LLAMA_CUBLAS=1 make approach is deprecated — use CMake with -DGGML_CUDA=ON. Azure’s NVIDIA GPU Driver Extension installs CUDA/GRID drivers on N-series VMs (NCasT4_v3, NC_A100_v4, NCads_H100_v5).

Containerize (official images). Pull ghcr.io/ggml-org/llama.cpp — variants include :server, :server-cuda (CUDA 12), :server-cuda13 (CUDA 13), :server-rocm, :server-vulkan, :server-musa, plus full/light variants. Pin an exact b#### tag; never use :latest in production.

docker run -d --gpus all -p 8080:8080 -v $PWD/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/phi-4.Q4_K_M.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers 999

docker-compose:

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    environment:
      - LLAMA_ARG_MODEL=/models/qwen2.5-7b-instruct-q4_k_m.gguf
      - LLAMA_ARG_N_GPU_LAYERS=999
    ports: ["8080:8080"]
    volumes: ["./models:/models"]
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]

Azure Container Instances (ACI): simplest managed container path; deploy the image with a GPU SKU (where available) or CPU, mount an Azure Files share for models. Good for spiky/intermittent workloads.

AKS with GPU node pools:

az aks nodepool add --resource-group rg --cluster-name aks \
  --name gpupool --node-count 1 --node-vm-size Standard_NC24ads_A100_v4 \
  --node-taints sku=gpu:NoSchedule
# Install the NVIDIA device plugin (or use the AKS GPU image), then deploy with
# resources.limits."nvidia.com/gpu": 1 and a matching toleration/nodeSelector.

Store models in an Azure Files/Blob-backed PersistentVolume so pods share them.

Networking / secure exposure. Terminate TLS and authenticate at Azure Application Gateway (with WAF) or an Azure Load Balancer / API Management in front of the server, keep llama-server on a private subnet/NSG, and never expose it directly — llama.cpp’s server has no built-in auth beyond an optional API key (--api-key).

Cost considerations. For steady low-concurrency chat, a single T4 (NCasT4_v3) is the cheapest GPU option; A100 (NC A100 v4) suits larger models or higher throughput. Spot pricing cuts costs ~70–80% (e.g., NC24ads A100 v4 from $3.673 to ~$0.68/hr) but gives only ~30 seconds of eviction notice, so use it only with checkpointing/failover. Reserved instances cut ~35–63% for steady workloads. Account for storage, egress, and (if using Azure ML compute) a ~25% surcharge over raw VM rates (e.g., NC96ads A100 v4 at $39.91/hr in Azure ML vs $31.93/hr for the raw VM).

5. Cloud Deployment — AWS

Instance selection.

  • CPU: c7g/c7gn (Graviton3, ARM) — llama.cpp’s NEON path makes Graviton very cost-effective for CPU inference; also c7i (Intel, with AVX-512) — good with a BLAS build. c7i.xlarge-class instances handle small quantized models.
  • GPU: g5 (NVIDIA A10G 24 GB; g5.xlarge ~$1.006/hr on-demand), g6 (NVIDIA L4, better inference-per-dollar), g6e (L40S), g4dn (T4 16 GB, cheapest, ~$0.10/hr spot), and p4d/p5 (A100/H100, 8-GPU only, e.g. p5.48xlarge ~$6.88/hr per H100 after the June 2025 44% price cut — overkill for llama.cpp except very large models). g5/g6 are the sweet spot for single-node GGUF serving.

Build with CUDA (GPU) or NEON (Graviton).

# GPU (use a Deep Learning AMI with drivers preinstalled)
sudo apt-get update && sudo apt-get install -y build-essential cmake git
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j $(nproc)
# Graviton (ARM) CPU: NEON is auto-detected; optionally add OpenBLAS
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build --config Release -j $(nproc)

On a g4dn.xlarge (T4, 16 GB) a 7B Q4_K_M at 8k context can OOM — drop to Q3_K_S or reduce -c. Use --cont-batching to raise requests/sec under concurrency (e.g., on an RTX 3090 continuous batching raised requests/sec from ~1.5 to ~2.5 at 4 concurrent connections).

Docker / ECS / EKS. Same official ghcr.io/ggml-org/llama.cpp:server-cuda image with --gpus all (requires the NVIDIA Container Toolkit). On ECS/Fargate, note Fargate has no GPU support, so GPU workloads need ECS on EC2 GPU instances; Fargate suits CPU-only inference. On EKS, add a GPU-enabled managed node group (g5/g6), install the NVIDIA device plugin, and request nvidia.com/gpu: 1 in the pod spec.

Autoscaling. Put llama-server behind an Application Load Balancer + target group with /health checks, and use an EC2 Auto Scaling Group or EKS Cluster Autoscaler/Karpenter scaling on GPU utilization or request-queue depth. Because model load time is significant, keep a warm minimum and use generous scale-in cooldowns.

S3 model storage patterns. Store GGUF files in S3; on instance/pod startup, pull to fast local NVMe (instance store) via aws s3 cp (or mount via Mountpoint for S3) before starting the server. This keeps AMIs/images small and decouples model versioning from compute. Cache to an EBS gp3 volume or the instance store to avoid re-downloading on restart.

Cost: spot vs on-demand. Spot saves 60–90% on g5/g6/g4dn but instances can be reclaimed with a 2-minute warning; ideal for stateless, load-balanced, autoscaled fleets with fast model reload from S3/EBS. Use on-demand or reserved/savings plans for a stable baseline and spot for burst capacity. Right-size: a single g6/g5 usually beats multiple smaller GPUs for a given model on cost-per-token because llama.cpp’s multi-GPU layer-split doesn’t add compute throughput.

6. Comparison & Decision Framework

llama.cpp vs vLLM vs TGI vs SGLang.

  • llama.cpp — best for single-user, edge, CPU, Apple Silicon, AMD-via-Vulkan, mixed CPU/GPU (models larger than VRAM), and maximum portability. Minimal dependencies, 1.5–8-bit quantization, runs almost anywhere. Historically weaker at high-concurrency serving, though continuous batching and 2026 kernel rewrites narrowed the gap.
  • vLLM — the high-throughput GPU serving standard; PagedAttention + continuous batching deliver large throughput gains. The original UC Berkeley benchmark (June 2023) reported vLLM at “14x–24x higher throughput than HF and 2.2x–2.5x higher throughput than TGI” (LLaMA-7B on A10G, LLaMA-13B on A100 40 GB, ShareGPT); a Nov 2025 arXiv study likewise found “up to 24x higher throughput than TGI under high-concurrency workloads.” GPU-only (CUDA, experimental ROCm/TPU). Choose for many concurrent users in production.
  • SGLang — emerging challenger, often beating vLLM on throughput/TTFT in 2025 benchmarks (e.g., ~920 vs ~870 tok/s at 10 concurrent users on an RTX 4090; ~2,850 vs ~2,400 tok/s on an A100 80 GB with DeepSeek-R1-32B); Linux-only, smaller community.
  • TGI (Hugging Face) — strong grammar/structured-output engine and HF-stack integration, but entered maintenance mode on December 11, 2025 (announced by HF’s Lysandre Jik: “text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks”) and the repo was archived read-only on March 21, 2026. HF now recommends vLLM or SGLang for new deployments.

Rule of thumb: ship a production API for concurrent users → vLLM (or SGLang); run locally, on a Mac, on CPU, on the edge, or need a model bigger than your VRAM → llama.cpp; already on TGI → keep it but plan migration. All four expose OpenAI-compatible APIs, so application code ports easily.

Known limitations of llama.cpp. (1) High-concurrency throughput trails vLLM/SGLang despite continuous batching. (2) Multi-GPU default (--split-mode layer) scales VRAM but not compute; row-split needs fast interconnect. (3) Server multimodal support is newer and less battle-tested than CLI. (4) Grammar sampling can hang on very long contexts with some dense models and adds ~8–15% overhead. (5) No prebuilt CUDA binaries from the core repo historically (use Docker images or build yourself); frequent b#### releases occasionally change server API/CLI flags (e.g., the 2026 --spec- rename), so pin versions. (6) The server’s built-in auth is minimal — put it behind a gateway.

Recommendations

  1. Start local on the exact hardware you’ll deploy on. On a Mac, brew install llama.cpp and run llama-cli -hf <repo> -ngl 99. On Linux/GPU, build with cmake -B build -DGGML_CUDA=ON. Validate tokens/sec with llama-bench before committing to a cloud SKU.
  2. Pick quantization by memory budget, then verify quality. Default to Q4_K_M; move up to Q5_K_M/Q6_K for small models or quality-critical use, down to IQ types with an imatrix only to fit tight memory. Always measure ppl/KL-divergence against the F16 baseline on your own prompts.
  3. For serving, decide by concurrency. ≤ a handful of simultaneous users, edge, CPU, or Mac → llama.cpp with --cont-batching --parallel N --flash-attn on. Dozens+ of concurrent GPU users → vLLM/SGLang. Benchmark your real workload; don’t trust generic tok/s numbers.
  4. Cloud sizing. Prototype on AWS g5.xlarge (~$1.006/hr) / Azure NCasT4_v3 (from $0.105/hr) (or CPU c7g/Fsv2 for light loads). Scale to A10G/L4 (g5/g6) or A100 (NC A100 v4, $3.673/hr) for larger models. Use the official pinned Docker image, store models in S3/Blob, front the server with an ALB/Application Gateway + TLS + API key, and use spot with checkpointed autoscaling for burst, on-demand/reserved for baseline.
  5. Production hygiene. Pin the b#### tag; keep a warm minimum instance (model load is slow); disable extreme KV quantization for tool-calling; add /health checks and Prometheus /metrics; and re-test CLI flags after upgrades (the CLI changes between builds).

Thresholds that change these recommendations: if sustained concurrency exceeds ~10–20 simultaneous requests per GPU, migrate the hot path to vLLM/SGLang. If your model no longer fits a single GPU’s VRAM even at Q4, either move to RPC/multi-GPU layer-split (accepting no compute speedup) or a larger single GPU. If p99 latency matters more than cost and acceptance rates are high, enable speculative decoding; if it slows you down, disable it.

Caveats

  • Fast-moving project. llama.cpp ships many builds per week with no semantic versioning; specific flags, defaults, and features (e.g., the 2026 --spec- speculative-decoding rename, GGUF-default imatrix output) change between builds. Treat exact flag names as version-dependent and check --help on your build.
  • Pricing is indicative and volatile. Cloud GPU rates cited (AWS g5.xlarge ~$1.006/hr, Azure NC24ads A100 v4 $3.673/hr, NC4as_T4_v3 from $0.105/hr, p5.48xlarge ~$6.88/hr per H100) are on-demand snapshots from 2025–2026 (some from third-party trackers) and vary by region, date, and commitment; confirm current AWS/Azure pricing pages before budgeting.
  • Benchmark figures are contextual. Throughput comparisons (llama.cpp vs vLLM/SGLang, the 14–24× vLLM-vs-HF/TGI figures, speculative-decoding speedups) depend heavily on model, quant, hardware, prompt/generation lengths, and concurrency; several are from vendor or community blogs. Run workload-specific benchmarks.
  • GitHub-stars and status figures are approximate mid-2026 snapshots (124K stars on Aug 14, 2026; some trackers report 118K+) and will have grown.
  • Some cited tutorials are third-party (Markaicode, Medium, vendor blogs) and may use outdated commands (e.g., LLAMA_CUBLAS, ghcr.io/ggerganov/... — now ghcr.io/ggml-org/...); the official ggml-org GitHub docs are authoritative.

Read the companion references:

Part of the Univrs research ecosystem: