#inference

GT4SD, an open-source library to accelerate hypothesis generation in the scientific discovery process.

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

No repository description published.

High-performance on-device LLM inference for React Native, powered by LiteRT-LM and Nitro Modules

Advanced CLI tool that scans your hardware and tells you exactly which LLM or sLLM models you can run locally, with full Ollama integration.

Distribute GGUF and safetensors weights through any OCI registry, with signatures verified before a byte moves. Daemonless, runs offline, one static binary.

vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm

NeLux: High-performance video processing for Python, powered by FFmpeg and PyTorch. Ultra-fast video decoding directly to tensors.

A collection of worker runtimes (with Dockerfiles) + worker functions, to be used by the gen-orchestrator.

A Polars plugin for JSON schema inference from string columns using genson-rs.

EdgeFirst TFLite for Rust

QNTX = Experiential ꩜ Learning ⌬ System ≡ Attestation +

Search engine

Multi-engine LLM benchmark & monitoring CLI for Apple Silicon

An out-of-tree vLLM plugin for Mobilint NPU runtime integration.

library to quickly export and deploy entire inference pipelines

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGUF on your exact hardware, self-calibrates, emits the exact llama.cpp command — or 'quantprobe auto' does it all. Falsification-tested laws; misses published at full size. pip install quantprobe

CSGLite is a lightweight local LLM inference platform. One command downloads, loads, and chats with models. It ships a web UI, OpenAI-compatible API, llama.cpp inference, resumable downloads, marketplace browsing, and one-click AI app and coding-agent setup—all in a single cross-platform binary.

No repository description published.

OramaCore is the complete runtime you need for your projects, answer engines, copilots, and search. It includes a fully-fledged full-text search engine, vector database, LLM interface, and many more utilities.

stanli, the Stan Language Interpreter: op-graph executor over precompiled stan-math kernels. Compile and sample Stan models with no C++ toolchain.

TurboQuant+ KV cache compression for vLLM. 3.8x smaller KV cache, same conversation quality. Fused CUDA kernels with automatic PyTorch fallback.