#inference

The verification layer for LLM serving — your check runs on every answer before it ships, with a signed, tamper-evident receipt for each decision and a distribution-free bound on wrong answers served. Proof over prediction.

Training-free KV cache compression via E8 lattice VQ. 2-bit KV that preserves retrieval (30/30 NIAH vs TurboQuant 0/30). Calibration-free, 9 architectures validated.

Harness Development Kit for Vertical Inference

Local inference daemon. One warm model, many consumers.

A simple implementation of the OWL2 RL Profile on top of RDFLib: it expands the graph with all possible triples that OWL RL defines. It can be used together with RDFLib to expand an RDFLib Graph object, or as a stand alone service with its own serialization.

Out-of-tree vLLM plugin and open format spec for NVFP4-CB / FP8-CB product-codebook weights — 2-6 bit-per-weight LLM quantization served on native Blackwell tensor cores.

AI-ready biological data I/O, validation, and tokenization engine

An advanced Java based inference engine for Large Language Models

该仓库未发布描述。

OpenVINO wrapper for .NET.

Free AI inference router for developers - 21 providers, OpenAI-compatible

Fused CUDA kernels for LLM inference - RMSNorm / RoPE / SwiGLU

Is your llama.cpp decode memory-bandwidth-bound? Find out in one command.

TypeScript SDK, CLI, agent, and relayer for Tangle AI Cloud — decentralized LLM inference

Use any AI model, locally or in the cloud, while tracking on-device token usage and collecting token telemetry data.

Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine in one command. No Electron, no Python, offline-first.

A superfast proxy and smart load-balancer for AI Inference — virtualize models, share local and provider back-ends, optimal caching, lock-in sampling parameters, debug message flow and obtain OTel metrics

ローカルLLM（mlx / mlx-vlm / llama.cpp / router）を OpenAI 互換 API として起動・管理する軽量サーバー

Complete python SDK, CLI, and API reference for integrating with the Pictograph.io platform.

该仓库未发布描述。