#inference

Port of OpenAI's Whisper model in C/C++

AI and cloud-native proxy server and framework

Calculate prices for calling LLM inference APIs.

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

Hundreds of models & providers. One command to find what runs on your hardware.

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Translation layer that maps any Kubernetes framework's Custom Resource Definitions (CRDs) into a standardized, generic structure.

TypeQL: Built for systems, not records

ncnn is a high-performance neural network inference framework optimized for the mobile platform

KEDA External gRPC Scaler for GPU workloads - native NVML metrics via DaemonSet, no Prometheus required

On-device Speech-to-Intent engine powered by deep learning

Next Generation Machine Learning, Statistics and Deep Learning in PURE Rust

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

Type-safe, real-time backend framework on your own Cloudflare account — Workers, Durable Objects, D1, R2, Queues. Convex-style DX, Vite-first.

Turn any computer or edge device into a command center for your computer vision projects.

Fast, flexible LLM inference

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

Cross-platform, customizable ML solutions for live and streaming media.

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

Fast inference engine for Transformer models

The One Dashboard every Voice Agent needs

Shared foundation for building MCP servers -- Streamable HTTP transport, OAuth 2.1, browser-based credential setup, and a shared embedding daemon.

Normalizing-flow enhanced sampling package for probabilistic inference in JAX

⚡ A seamless integration of HuggingFace Transformers & Diffusers with RBLN SDK for efficient inference on RBLN NPUs.

PyPI capacity-planning CLI for LLM deployment. pip install chimeraforge.

Unified LLM API, structured data outputs with LLMs, and agent SDK - in PHP