vLLM

By vLLM project ยท Inference engines and runtimes

About vLLM

A high-throughput serving engine for language models on GPUs, often used to share one strong machine with a team.

Details

  • Open source

    Yes

  • Local API server

    Yes

  • Runs on Linux

    Yes

Not enough info yet on works offline after setup, runs on macOS, runs on Windows. Check the tool's own site.

Listing written by LaptopMind, checked 2026-11-04. Report a correction or request removal. Back to inference engines and runtimes.

More tools in inference engines and runtimes

SGLang

by SGLang project

A fast serving framework for language and vision models on GPUs, with caching that helps when prompts share long prefixes.

  • Open source
  • Local API server

TensorRT-LLM

by NVIDIA

NVIDIA's library for compiling and running language models with kernels tuned for its GPUs.

  • Open source

ONNX Runtime

by Microsoft

A cross-platform runtime for models in the ONNX format, with backends for CPUs, GPUs and NPUs on many devices.

  • Open source
  • Runs on macOS
  • Runs on Windows

OpenVINO

by Intel

Intel's toolkit for optimizing and running models on Intel CPUs, integrated graphics and NPUs.

  • Open source

ExLlamaV2

by turboderp

A fast inference library for running quantized language models on consumer NVIDIA GPUs.

  • Open source

Core ML

by Apple

Apple's framework for running models on device across the CPU, GPU and Neural Engine in apps for Apple platforms.

  • Runs on macOS