#model-serving

A high-throughput and memory-efficient inference and serving engine for LLMs

The simplest way to serve AI/ML models in production

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Community maintained hardware plugin for vLLM on Huawei Ascend

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.