#inference

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

A Datacenter Scale Distributed Inference Serving Framework

A high-throughput and memory-efficient inference and serving engine for LLMs

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

Open ABI and FFI for Machine Learning Systems

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

PyTorch native quantization for training and inference

SGLang is a high-performance serving framework for large language models and multimodal models.

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Community maintained hardware plugin for vLLM on Huawei Ascend

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Communicate with an LLM provider using a single interface

The AI Toolkit for TypeScript. From the creators of Next.js, the AI SDK is a free open-source library for building AI-powered applications and agents

Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.

Offline optimization of your disaggregated Dynamo graph

Model Express is a Rust-based component meant to be placed next to existing model inference systems to speed up their startup times and improve overall performance.

Use Hugging Face with JavaScript

Adds static typing to JavaScript to improve developer productivity and code quality.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Fast ML inference & training for ONNX models in Rust

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT

The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

A visualization grammar.

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM