Inference engines and runtimes: local AI tools

Inference engines and runtimes are the layer that turns a model file into fast arithmetic on a particular chip. Where a general runner aims to work everywhere, these projects are tuned for specific hardware: NVIDIA data center and gaming GPUs, Intel processors and NPUs, Apple silicon, or the mix of devices a cross-platform app has to support.

Most individuals never touch this layer directly. It matters when you serve one model to several people from a single strong machine, build your own app that ships a model to users, or need every bit of speed from the hardware you already own. Engines built for serving handle many requests at once, batching them together so a single GPU can answer a small team without each person waiting in line.

The trade-off is setup effort and narrower support. Serving engines usually expect Linux and a recent NVIDIA GPU, and some need models converted or compiled for the exact card before they run. Runtimes from chip makers run fastest on that maker's hardware, and the speed gains rarely carry over to other devices. Read the supported hardware list before anything else.

Check which model formats and quantization methods an engine accepts. Some read standard weights directly, some need their own compiled format, and quantized files made for one engine often do not load in another. Converting is possible but adds steps, and a model can behave slightly differently after conversion, so compare outputs on a few prompts you know well.

Featured local AI tools: inference engines and runtimes

SponsoredInference engines and runtimes This position is open Put your tool at the top of the inference engines and runtimes page. Paid spots are always labeled Sponsored. From $99 a month. Get featured

What a featured listing costs · from $99 a month

7 local AI tools in inference engines and runtimes

vLLM

by vLLM project

A high-throughput serving engine for language models on GPUs, often used to share one strong machine with a team.

Open source
Local API server
Runs on Linux

SGLang

by SGLang project

A fast serving framework for language and vision models on GPUs, with caching that helps when prompts share long prefixes.

Open source
Local API server

TensorRT-LLM

by NVIDIA

NVIDIA's library for compiling and running language models with kernels tuned for its GPUs.

Open source

ONNX Runtime

by Microsoft

A cross-platform runtime for models in the ONNX format, with backends for CPUs, GPUs and NPUs on many devices.

Open source
Runs on macOS
Runs on Windows

OpenVINO

by Intel

Intel's toolkit for optimizing and running models on Intel CPUs, integrated graphics and NPUs.

Open source

ExLlamaV2

by turboderp

A fast inference library for running quantized language models on consumer NVIDIA GPUs.

Open source

Core ML

by Apple

Apple's framework for running models on device across the CPU, GPU and Neural Engine in apps for Apple platforms.

Runs on macOS

Look at how the engine is operated over time. A serving engine is a long-running service with logs, memory limits and updates, much like a database. If you run one for a team, plan who restarts it, who updates drivers and how you notice when it stops answering.

Measure before and after any change. Swapping engines, quantization methods or driver versions can speed things up on one workload and slow them down on another, and the difference often depends on prompt length and how many requests arrive at once. Time a handful of real requests from your own use, keep the numbers with the settings that produced them, and only switch when the gain holds up on your traffic rather than on a published chart.

For app builders, platform runtimes on phones and laptops can run small models on the device's neural hardware with low power use. That is useful for features such as on-device transcription or summaries, where sending data to a server would raise privacy questions.

Questions about inference engines and runtimes

Do I need an inference engine to use local AI?

Not for personal use. Desktop apps and runners already include an engine. Dedicated engines matter when you serve a model to several people, build your own product, or want to tune performance on specific hardware.

Why does an engine need my model converted first?

Some engines compile the model for your exact GPU or use their own optimized format, which is part of how they gain speed. The cost is an extra conversion step, and the converted files usually work only with that engine.

Will an engine made for one chip maker run on another's hardware?

Often not, or not with the same benefit. Vendor runtimes are tuned for their own processors. Cross-platform runtimes support several backends, but each backend has its own limits, so check the supported devices list.

How do I share one GPU machine with my team?

Run a serving engine or runner on that machine, put it behind authentication on your network, and point everyone's apps at its address. Watch memory use and set limits, since many simultaneous long requests can fill a card quickly.

Inference engines and runtimes