#gguf

Quantize an LLM and check it still refuses what it should — a GPU-aware quantization CLI that measures the safety drift of the quantization it just performed.

A rust gguf library

Distribute GGUF and safetensors weights through any OCI registry, with signatures verified before a byte moves. Daemonless, runs offline, one static binary.

Go with your own intelligence - Write Go applications that directly integrate llama.cpp for local inference using hardware acceleration on Linux, macOS, Windows, & WebAssembly.

ggml speech-to-text inference for 16+ model families

.NET library for on-demand local AI model inference — zero bundled models, lazy loading, hardware-aware GPU/CPU selection, 10 task types including embeddings, generation, vision, and audio.

Ai training toolkit

Native K-quant support for MLX, with a quantization and fine-tuning toolchain for Apple Silicon

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGUF on your exact hardware, self-calibrates, emits the exact llama.cpp command — or 'quantprobe auto' does it all. Falsification-tested laws; misses published at full size. pip install quantprobe

Agentic Runtime

In-JVM small language model inference

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

Pure-Go, no-cgo local LLM inference — run Gemma, Qwen, Llama and friends from safetensors or GGUF in a single static binary.