How to Install llama-cpp-python

Install llama-cpp-python from an official wheel index to avoid compiling it. A bare pip install llama-cpp-python builds from source; GPU support depends on the selected backend, with Metal enabled by default on macOS.

Choose a backend

Prerequisites

Install uv using the uv installation guide, then create a virtual environment in an empty directory. --seed includes pip for the alternative commands.

uv venv --python 3.12 --seed

Expect Creating virtual environment with seed packages at: .venv. Activate the environment for your shell before continuing, then choose one backend.

Install from the CPU index

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Expect an Installed summary including llama-cpp-python. For pip, replace uv pip install with pip install in the wheel and source commands; expect Successfully installed.

--only-binary=llama-cpp-python makes a missing wheel fail instead of silently compiling. --force-reinstall replaces an existing backend. If no wheel matches, build from source. On macOS, general wheels and default source builds include Metal; for damaged archives, use the Metal source fallback.

Install CUDA for an NVIDIA GPU

Choose an index from the official CUDA wheel matrix. Compute capability identifies the GPU architecture; look up the model in NVIDIA’s GPU tables.

Current Linux CUDA wheels require glibc 2.35 or newer, as recorded in the published wheel tags.

Install a compatible NVIDIA driver and matching CUDA runtime libraries, including cuBLAS; these wheels do not bundle them. Installing the matching CUDA Toolkit supplies those libraries.

The CUDA Version in nvidia-smi describes driver support, not an installed toolkit. The documented compute-capability floor does not guarantee that an older toolkit supports a newer GPU; check the CUDA build requirements.

For a CUDA 12.5-compatible system:

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu125

Expect installation from a wheel. Replace cu125 with the selected suffix. If no compatible wheel exists, install the compiler prerequisites and a toolkit that supports the GPU, ensure nvcc is on PATH, and rebuild:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir -C cmake.args=-DGGML_CUDA=on

Expect a CUDA compilation followed by installation.

Install Metal on Apple Silicon

Use arm64-native Python on macOS 11 or later. Confirm the interpreter architecture:

python -c "import platform; print(platform.machine())"

Expect arm64. If it prints x86_64, recreate the environment using arm64 Python before installing:

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

A valid wheel installs without compilation. If uv reports ZIP file contains trailing contents or Bad uncompressed size, or pip reports Bad CRC-32, the published archive is damaged; use the source fallback with Xcode command-line tools installed:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir -C cmake.args=-DGGML_METAL=on

Expect Built llama-cpp-python followed by installation.

Install other GPU backends

Use the CPU wheel command with the index suffix for the selected backend. Expect installation from a wheel.

Check AMD’s compatibility matrix before choosing ROCm. For source builds, follow the upstream backend instructions; Intel GPUs using SYCL need the oneAPI setup.

Save the backend in a uv project

For an existing uv project, add this configuration to pyproject.toml before adding the dependency. Replace cpu in the URL with the compatible backend suffix; the backend’s platform and runtime requirements still apply.

[[tool.uv.index]]
name = "llama-cpp-python"
url = "https://abetlen.github.io/llama-cpp-python/whl/cpu"
explicit = true

[tool.uv.sources]
llama-cpp-python = { index = "llama-cpp-python" }

explicit = true reserves this index for the named package. Add the dependency with source builds disabled:

uv add llama-cpp-python --no-build-package llama-cpp-python

Expect an installation summary and updates to pyproject.toml and uv.lock. Commit both files to preserve the backend choice and resolved version.

Verify the installation

python -c "from llama_cpp import llama_supports_gpu_offload; print('GPU offload supported:', llama_supports_gpu_offload())"

Expect GPU offload supported: False for Linux/Windows CPU builds or True for an available GPU backend, including Metal in macOS builds. This checks backend availability; it does not prove that a model uses the GPU. If import reports a missing CUDA, HIP, or Vulkan library, install the matching runtime; if it reports an incompatible architecture, select a wheel matching the interpreter and OS.

Resolve installation failures

Source builds need GCC or Clang on Linux, Xcode command-line tools on macOS, or Visual Studio Build Tools with C++ on Windows. The isolated build installs CMake. If it requests missing clang, install Clang or select a supported compiler.

For the default source build:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir

Expect Built llama-cpp-python followed by installation. Keep all three flags in source commands: --no-binary selects source, --force-reinstall replaces the installed package, and --no-cache-dir prevents cached builds from being reused.

For CUDA and Metal wheels, the README documents Python 3.10–3.12. Generic py3-none wheel tags do not enforce that range; check the index for the release and platform needed when using another Python version.