ExLlamaV2

By turboderp ยท Inference engines and runtimes

About ExLlamaV2

A fast inference library for running quantized language models on consumer NVIDIA GPUs.

Details

  • Open source

    Yes

Not enough info yet on works offline after setup, local API server, runs on macOS, runs on Windows, runs on Linux. Check the tool's own site.

Listing written by LaptopMind, checked 2026-11-04. Report a correction or request removal. Back to inference engines and runtimes.

More tools in inference engines and runtimes

vLLM

by vLLM project

A high-throughput serving engine for language models on GPUs, often used to share one strong machine with a team.

  • Open source
  • Local API server
  • Runs on Linux

SGLang

by SGLang project

A fast serving framework for language and vision models on GPUs, with caching that helps when prompts share long prefixes.

  • Open source
  • Local API server

TensorRT-LLM

by NVIDIA

NVIDIA's library for compiling and running language models with kernels tuned for its GPUs.

  • Open source

ONNX Runtime

by Microsoft

A cross-platform runtime for models in the ONNX format, with backends for CPUs, GPUs and NPUs on many devices.

  • Open source
  • Runs on macOS
  • Runs on Windows

OpenVINO

by Intel

Intel's toolkit for optimizing and running models on Intel CPUs, integrated graphics and NPUs.

  • Open source

Core ML

by Apple

Apple's framework for running models on device across the CPU, GPU and Neural Engine in apps for Apple platforms.

  • Runs on macOS