llama.cpp

By ggml-org ยท Local model runners

About llama.cpp

The C and C++ engine behind many local AI apps, for running quantized GGUF models on CPUs, GPUs and Apple silicon.

llama.cpp is an open-source project for running language models in plain C and C++. It introduced and maintains the GGUF file format, which stores a model and its settings in one file and supports quantized versions that need much less memory than the original weights.

Many desktop apps and runners use it under the hood. Used directly, it gives you command-line tools for chatting, converting and quantizing models, plus a server that answers requests from other programs. It can split a model between the CPU and a graphics card when the model does not fit in video memory.

It suits developers and people who want the most control over how a model runs. Building it yourself lets you pick the right backend for your hardware, while prebuilt releases are available for common systems.

Details

  • Open source

    Yes

  • Works offline after setup

    Yes

  • Local API server

    Yes

  • Runs on macOS

    Yes

  • Runs on Windows

    Yes

  • Runs on Linux

    Yes

Listing written by LaptopMind, checked 2026-11-04. Report a correction or request removal. Back to local model runners.

More tools in local model runners

Ollama

A command-line runner and local server that downloads open models by name and serves them to other apps on your machine.

  • Open source
  • Works offline after setup
  • Local API server

llamafile

by Mozilla

Packs a model and its runtime into one executable file that runs on most operating systems without an install step.

  • Open source
  • Works offline after setup
  • Local API server

LocalAI

A self-hosted server that runs text, image and audio models locally behind an API shaped like common hosted ones.

  • Open source
  • Works offline after setup
  • Local API server

KoboldCpp

by LostRuins

A single-file runner for GGUF models with a built-in web interface, often used for long-form writing and story sessions.

  • Open source
  • Works offline after setup
  • Local API server

MLX

by Apple

An array framework from Apple research for running and training models efficiently on Apple silicon memory.

  • Open source
  • Runs on macOS

MLC LLM

by MLC AI

Compiles language models to run natively on laptops, phones and in the browser through one deployment workflow.

  • Open source