#inference

A collection of worker runtimes (with Dockerfiles) + worker functions, to be used by the gen-orchestrator.

Hardware-aware CLI that detects NPUs/GPUs, finds compatible GGUF models, and recommends optimal llama.cpp inference configs

Training-free KV cache compression via E8 lattice VQ. 2-bit KV that preserves retrieval (30/30 NIAH vs TurboQuant 0/30). Calibration-free, 9 architectures validated.

fak — the Fused Agent Kernel: one Go binary for AI agent loops. Wrap Claude Code/Codex/Cursor, keep long sessions cache-efficient, route work per call, run local GGUF models, and adjudicate tool calls.

Multi-engine LLM benchmark & monitoring CLI for Apple Silicon

Advancing the frontier of efficient AI

Faster Whisper transcription with CTranslate2

No repository description published.

Run, quantize, and fine-tune LLMs on Apple Silicon. Pure Rust, no Python, no CUDA, no ONNX

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine in one command. No Electron, no Python, offline-first.

A superfast proxy and smart load-balancer for AI Inference — virtualize models, share local and provider back-ends, optimal caching, lock-in sampling parameters, debug message flow and obtain OTel metrics

Search engine

ローカルLLM（mlx / mlx-vlm / llama.cpp / router）を OpenAI 互換 API として起動・管理する軽量サーバー

Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

CSGLite is a lightweight local LLM runner for the CSGHub platform. One command downloads, loads, and chats with models. It ships a web UI, OpenAI-compatible API, llama.cpp inference, resumable downloads, marketplace browsing, and one-click AI app and coding-agent setup—all in a single cross-platform binary.

Lightweight and friendly .NET library for realizing modern Semantic Web applications (OWL2, SWRL)

Provides building blocks to integrate with AI / LLM providers and built-in, common, providers.