#inference

Communicate with an LLM provider using a single interface

PyTorch native quantization and sparsity for training and inference

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Fast, Flexible and Portable Structured Generation

SGLang is a high-performance serving framework for large language models and multimodal models.

TypeScript-first schema validation with static type inference

Cross-platform, customizable ML solutions for live and streaming media.

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Nvidia GPU exporter for prometheus using nvidia-smi binary

Translation layer that maps any Kubernetes framework's Custom Resource Definitions (CRDs) into a standardized, generic structure.

Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

Fast inference engine for Transformer models

Hundreds of models & providers. One command to find what runs on your hardware.

Turn any computer or edge device into a command center for your computer vision projects.

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.