Local model runners

A local model runner is the engine you drive from a terminal. You name a model, the runner fetches it, loads it with sensible settings and either opens a chat in the terminal or starts a small server that answers requests from other programs. Many desktop apps are built on one of these runners, so learning one gives you a clearer view of what the friendly apps are doing underneath.

The server is the main reason people use a runner directly. With a model loaded behind a local API, a chat interface, a coding extension, a document tool and your own scripts can all send it work, and you only pay the memory cost once. Several runners copy the request format of widely used hosted APIs, so tools written for those services often work locally after changing one address.

Runners differ in how much they decide for you. Some keep a curated library and hide every setting until you ask for it. Others expose every option for loading, splitting a model between CPU and GPU, and controlling how text is sampled. A few package the model and the runtime into one file you can copy to another machine and run without installing anything.

Hardware support is the next thing to check. Most runners handle NVIDIA graphics cards and Apple silicon well. Support for AMD and Intel GPUs varies between projects and versions, and the CPU fallback is always slower. If you are on a less common setup, read the project's install notes before you commit an afternoon to it.

Featured local AI tools: local model runners

SponsoredLocal model runners This position is open Put your tool at the top of the local model runners page. Paid spots are always labeled Sponsored. From $99 a month. Get featured

What a featured listing costs ยท from $99 a month

7 local AI tools in local model runners

Ollama

A command-line runner and local server that downloads open models by name and serves them to other apps on your machine.

  • Open source
  • Works offline after setup
  • Local API server

llama.cpp

by ggml-org

The C and C++ engine behind many local AI apps, for running quantized GGUF models on CPUs, GPUs and Apple silicon.

  • Open source
  • Works offline after setup
  • Local API server

llamafile

by Mozilla

Packs a model and its runtime into one executable file that runs on most operating systems without an install step.

  • Open source
  • Works offline after setup
  • Local API server

LocalAI

A self-hosted server that runs text, image and audio models locally behind an API shaped like common hosted ones.

  • Open source
  • Works offline after setup
  • Local API server

KoboldCpp

by LostRuins

A single-file runner for GGUF models with a built-in web interface, often used for long-form writing and story sessions.

  • Open source
  • Works offline after setup
  • Local API server

MLX

by Apple

An array framework from Apple research for running and training models efficiently on Apple silicon memory.

  • Open source
  • Runs on macOS

MLC LLM

by MLC AI

Compiles language models to run natively on laptops, phones and in the browser through one deployment workflow.

  • Open source

Because these are developer tools, think about where the server listens. By default most bind to your own machine only. Opening one to your local network is handy for sharing a desktop GPU with a laptop, but it also means anyone on that network can use the model, so add authentication or a firewall rule first.

Keep an eye on disk space as well. Runners cache every model you pull, and model files are measured in gigabytes, so a week of trying things can fill a laptop drive. Most runners have a command to list and remove models; use it regularly, and move the model folder to a larger drive if your system lets you choose where it lives.

As with every local tool, the runner's license is separate from the model's. Model pages list the terms, and some models ask you to accept conditions before downloading.

Questions about local model runners

What is the difference between a runner and a desktop app?

A runner is usually a command-line tool and a local server, without much interface. A desktop app adds a chat window and settings screens, and is often built on top of a runner. Many people use both: the runner in the background and an app or editor in front.

Is it safe to expose a local model server on my network?

It can be, with care. Out of the box most runners listen only on your own machine. If you open one to your network, anyone on it can send requests, so add a password or proxy and keep the runner updated.

Do runners work on computers without a graphics card?

Yes. They fall back to the CPU, which works for small models and short answers. Expect slower replies, and choose smaller quantized files so the model fits in normal RAM.

Can I run several models at once?

Most runners can keep more than one model loaded if memory allows, and some unload idle models automatically. Each loaded model takes its own share of memory, so on a laptop it is usually better to run one at a time.

Related categories