ONNX Runtime

By Microsoft ยท Inference engines and runtimes

About ONNX Runtime

A cross-platform runtime for models in the ONNX format, with backends for CPUs, GPUs and NPUs on many devices.

Details

  • Open source

    Yes

  • Runs on macOS

    Yes

  • Runs on Windows

    Yes

  • Runs on Linux

    Yes

Not enough info yet on works offline after setup, local API server. Check the tool's own site.

Listing written by LaptopMind, checked 2026-11-04. Report a correction or request removal. Back to inference engines and runtimes.

More tools in inference engines and runtimes

vLLM

by vLLM project

A high-throughput serving engine for language models on GPUs, often used to share one strong machine with a team.

  • Open source
  • Local API server
  • Runs on Linux

SGLang

by SGLang project

A fast serving framework for language and vision models on GPUs, with caching that helps when prompts share long prefixes.

  • Open source
  • Local API server

TensorRT-LLM

by NVIDIA

NVIDIA's library for compiling and running language models with kernels tuned for its GPUs.

  • Open source

OpenVINO

by Intel

Intel's toolkit for optimizing and running models on Intel CPUs, integrated graphics and NPUs.

  • Open source

ExLlamaV2

by turboderp

A fast inference library for running quantized language models on consumer NVIDIA GPUs.

  • Open source

Core ML

by Apple

Apple's framework for running models on device across the CPU, GPU and Neural Engine in apps for Apple platforms.

  • Runs on macOS