llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

When to use llama.cpp

Use llama.cpp when:

• Running on CPU-only machines

• Deploying on Apple Silicon (M1/M2/M3/M4)

• Using AMD or Intel GPUs (no CUDA)

• Edge deployment (Raspberry Pi, embedded systems)

• Need simple deployment without Docker/Python

Use TensorRT-LLM instead when:

• Have NVIDIA GPUs (A100/H100)

• Need maximum throughput (100K+ tok/s)

• Running in datacenter with CUDA

Use vLLM instead when:

• Have NVIDIA GPUs

• Need Python-first API

• Want PagedAttention

Quick start

Installation

# macOS/Linux

brew install llama.cpp

# Or build from source

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

make

# With Metal (Apple Silicon)

make LLAMA_METAL=1

# With CUDA (NVIDIA)

make LLAMA_CUDA=1

# With ROCm (AMD)

make LLAMA_HIP=1

Download model

# Download from HuggingFace (GGUF format)

huggingface-cli download \

    TheBloke/Llama-2-7B-Chat-GGUF \

    llama-2-7b-chat.Q4_K_M.gguf \

    --local-dir models/

# Or convert from HuggingFace

python convert_hf_to_gguf.py models/llama-2-7b-chat/

Run inference

# Simple chat

./llama-cli \

    -m models/llama-2-7b-chat.Q4_K_M.gguf \

    -p "Explain quantum computing" \

    -n 256  # Max tokens

# Interactive chat

./llama-cli \

    -m models/llama-2-7b-chat.Q4_K_M.gguf \

    --interactive

Server mode

# Start OpenAI-compatible server

./llama-server \

    -m models/llama-2-7b-chat.Q4_K_M.gguf \

    --host 0.0.0.0 \

    --port 8080 \

    -ngl 32  # Offload 32 layers to GPU

# Client request

curl http://localhost:8080/v1/chat/completions \

  -H "Content-Type: application/json" \

  -d '{

    "model": "llama-2-7b-chat",

    "messages": [{"role": "user", "content": "Hello!"}],

    "temperature": 0.7,

    "max_tokens": 100

  }'

Quantization formats

GGUF format overview

Format

Bits

Size (7B)

Speed

Quality

Use Case

Q4_K_M

4.5

4.1 GB

Fast

Good

Recommended default

Q4_K_S

4.3

3.9 GB

Faster

Lower

Speed critical

Q5_K_M

5.5

4.8 GB

Medium

Better

Quality critical

Q6_K

6.5

5.5 GB

Slower

Best

Maximum quality

Q8_0

8.0

7.0 GB

Slow

Excellent

Minimal degradation

Q2_K

2.5

2.7 GB

Fastest

Poor

Testing only

Choosing quantization

# General use (balanced)

Q4_K_M  # 4-bit, medium quality

# Maximum speed (more degradation)

Q2_K or Q3_K_M

# Maximum quality (slower)

Q6_K or Q8_0

# Very large models (70B, 405B)

Q3_K_M or Q4_K_S  # Lower bits to fit in memory

Hardware acceleration

Apple Silicon (Metal)

# Build with Metal

make LLAMA_METAL=1

# Run with GPU acceleration (automatic)

./llama-cli -m model.gguf -ngl 999  # Offload all layers

# Performance: M3 Max 40-60 tokens/sec (Llama 2-7B Q4_K_M)

NVIDIA GPUs (CUDA)

# Build with CUDA

make LLAMA_CUDA=1

# Offload layers to GPU

./llama-cli -m model.gguf -ngl 35  # Offload 35/40 layers

# Hybrid CPU+GPU for large models

./llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20  # GPU: 20 layers, CPU: rest

AMD GPUs (ROCm)

# Build with ROCm

make LLAMA_HIP=1

# Run with AMD GPU

./llama-cli -m model.gguf -ngl 999

Common patterns

Batch processing

# Process multiple prompts from file

cat prompts.txt | ./llama-cli \

    -m model.gguf \

    --batch-size 512 \

    -n 100

Constrained generation

# JSON output with grammar

./llama-cli \

    -m model.gguf \

    -p "Generate a person: " \

    --grammar-file grammars/json.gbnf

# Outputs valid JSON only

Context size

# Increase context (default 512)

./llama-cli \

    -m model.gguf \

    -c 4096  # 4K context window

# Very long context (if model supports)

./llama-cli -m model.gguf -c 32768  # 32K context

Performance benchmarks

CPU performance (Llama 2-7B Q4_K_M)

CPU

Threads

Speed

Cost

Apple M3 Max

16

50 tok/s

$0 (local)

AMD Ryzen 9 7950X

32

35 tok/s

$0.50/hour

Intel i9-13900K

32

30 tok/s

$0.40/hour

AWS c7i.16xlarge

64

40 tok/s

$2.88/hour

GPU acceleration (Llama 2-7B Q4_K_M)

GPU

Speed

vs CPU

Cost

NVIDIA RTX 4090

120 tok/s

3-4×

$0 (local)

NVIDIA A10

80 tok/s

2-3×

$1.00/hour

AMD MI250

70 tok/s

2×

$2.00/hour

Apple M3 Max (Metal)

50 tok/s

~Same

$0 (local)

Supported models

LLaMA family:

• Llama 2 (7B, 13B, 70B)

• Llama 3 (8B, 70B, 405B)

• Code Llama

Mistral family:

• Mistral 7B

• Mixtral 8x7B, 8x22B

Other:

• Falcon, BLOOM, GPT-J

• Phi-3, Gemma, Qwen

• LLaVA (vision), Whisper (audio)

Find models: https://huggingface.co/models?library=gguf

References

• Quantization Guide - GGUF formats, conversion, quality comparison

• Server Deployment - API endpoints, Docker, monitoring

• Optimization - Performance tuning, hybrid CPU+GPU

Resources

• GitHub: https://github.com/ggerganov/llama.cpp

• Models: https://huggingface.co/models?library=gguf

• Discord: https://discord.gg/llama-cpp