#llm-inference

A single interface in Go to get inference from multiple llm/ai providers using their official SDKs

Training-free KV cache compression via E8 lattice VQ. 2-bit KV that preserves retrieval (30/30 NIAH vs TurboQuant 0/30). Calibration-free, 9 architectures validated.

Out-of-tree vLLM plugin and open format spec for NVFP4-CB / FP8-CB product-codebook weights — 2-6 bit-per-weight LLM quantization served on native Blackwell tensor cores.

LLMs as Copilots for Theorem Proving in Lean

An imperative command-line-interface for AI workload orchestration

Animica — post-quantum (ML-DSA-65) proof-of-work Layer-1 blockchain by Alexander Lehman: Python smart contracts, ANM-native L2 rollup, mining pool, free OpenAI-compatible AI API, x402 pay-per-call agent APIs, MCP server. Monorepo, Apache-2.0.

LLM Tool implementations for Golang

A CLI that makes it easy way to find, run, and manage local LLMs.

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

The blood-brain barrier for autonomous agents. A context-aware LLM governance proxy that enforces credential starvation — identity-verified, provider-routed, cost-tracked, and audit-logged.

Run the real PyTorch ecosystem on device, not reimplementing it

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets