Maven · PyPI
ray-project/ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Apache-2.0Aug 5, 2026 · metrics 2.10.0
All tags
Catalogue tag
#llm-inference
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
35 records
PyPI · crates.io
ai-dynamo/dynamo
A Datacenter Scale Distributed Inference Serving Framework
Custom licenseAug 28, 2026 · metrics 2.10.0
PyPI
kvcache-ai/Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Apache-2.0Aug 12, 2026 · metrics 2.10.0
PyPI
flashinfer-ai/flashinfer
FlashInfer: Kernel Library for LLM Serving
Apache-2.0Aug 27, 2026 · metrics 2.10.0
Go · PyPI
kserve/kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
Apache-2.0Aug 28, 2026 · metrics 2.10.0
PyPI
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Apache-2.0Aug 12, 2026 · metrics 2.10.0
PyPI
gpustack/gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Apache-2.0Aug 2, 2026 · metrics 2.10.0
PyPI
lemonade-sdk/lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
Apache-2.0Aug 28, 2026 · metrics 2.10.0
PyPI
monocle2ai/monocle
Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.
Apache-2.0Aug 28, 2026 · metrics 2.10.0
PyPI
Lightning-AI/litgpt
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Apache-2.0Aug 27, 2026 · metrics 2.10.0
npm
Nano-Collective/nanocoder
An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.
Custom licenseAug 25, 2026 · metrics 2.10.0
npm · crates.io
ruvnet/RuVector
RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.
MITAug 22, 2026 · metrics 2.10.0
Packagist
neuron-core/neuron-ai
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that can interact with your data.
MITJul 15, 2026 · metrics 2.10.0
Go · npm
matrixhub-ai/matrixhub
An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.
Apache-2.0Jul 17, 2026 · metrics 2.10.0
Go · npm
ome-projects/ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
Apache-2.0Jul 21, 2026 · metrics 2.10.0
PyPI
Nayjest/lm-proxy
OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch). Lightweight, extensible Python/FastAPI—use as library or standalone service.
MITAug 26, 2026 · metrics 2.10.0
PyPI
algorithmicsuperintelligence/openevolve
Open-source implementation of AlphaEvolve
Apache-2.0Aug 28, 2026 · metrics 2.10.0
PyPI
Sahil170595/Chimeraforge
PyPI capacity-planning CLI for LLM deployment. pip install chimeraforge.
MITAug 22, 2026 · metrics 2.10.0
npm
webgptorg/promptbook
Turn your company's scattered knowledge into AI ready Books ✨
Custom licenseSep 6, 2026 · metrics 2.10.0
npm
requestyai/ai-sdk-requesty
The Requesty integration package for the Vercel AI SDK.
Apache-2.0Jul 22, 2026 · metrics 2.10.0
PyPI
asher/mlx-kquant
Native K-quant support for MLX, with a quantization and fine-tuning toolchain for Apple Silicon
MITAug 23, 2026 · metrics 2.10.0
Go · PyPI
anthony-chaudhary/fak
fak — the Fused Agent Kernel: one Go binary for AI agent loops. Wrap Claude Code/Codex/Cursor, keep long sessions cache-efficient, route work per call, run local GGUF models, and adjudicate tool calls.
Apache-2.0Jul 17, 2026 · metrics 2.10.0
PyPI
RobTand/gridbook
Out-of-tree vLLM plugin and open format spec for NVFP4-CB / FP8-CB product-codebook weights — 2-6 bit-per-weight LLM quantization served on native Blackwell tensor cores.
Apache-2.0Aug 15, 2026 · metrics 2.10.0
Go
flexigpt/inference-go
A single interface in Go to get inference from multiple llm/ai providers using their official SDKs
MITAug 3, 2026 · metrics 2.10.0
PyPI
jagmarques/nexusquant
Training-free KV cache compression via E8 lattice VQ. 2-bit KV that preserves retrieval (30/30 NIAH vs TurboQuant 0/30). Calibration-free, 9 architectures validated.
Custom licenseJul 16, 2026 · metrics 2.10.0
PyPI · crates.io
theoddden/Terradev
An imperative command-line-interface for AI workload orchestration
Apache-2.0Aug 23, 2026 · metrics 2.10.0
—
eastriverlee/LLM.swift
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
MITJul 28, 2026 · metrics 2.10.0
—
lean-dojo/LeanCopilot
LLMs as Copilots for Theorem Proving in Lean
MITAug 4, 2026 · metrics 2.10.0
PyPI
ai-hypercomputer/jetstream
JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).
Apache-2.0Jul 18, 2026 · metrics 2.10.0
Go
flexigpt/llmtools-go
LLM Tool implementations for Golang
MITAug 20, 2026 · metrics 2.10.0