PyPI · npm
Arize-ai/phoenix
AI Observability & Evaluation
Custom licenseAug 28, 2026 · metrics 2.10.0
All tags
Catalogue tag
#llm-evaluation
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
38 records
PyPI · npm
mlflow/mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Apache-2.0Sep 16, 2026 · metrics 2.10.0
npm · PyPI
comet-ml/opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Apache-2.0Sep 16, 2026 · metrics 2.10.0
PyPI
Giskard-AI/giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
Apache-2.0Aug 28, 2026 · metrics 2.10.0
npm
promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
MITSep 16, 2026 · metrics 2.10.0
PyPI · npm
NVIDIA/garak
the LLM vulnerability scanner
Apache-2.0Oct 19, 2026 · metrics 2.10.0
npm · crates.io
langfuse/langfuse
🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Custom licenseSep 16, 2026 · metrics 2.10.0
PyPI
Q00/ouroboros
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
MITSep 18, 2026 · metrics 2.10.0
PyPI · npm
NuGuardAI/nuguard
AI red-teaming tool and LLM security framework to evaluate agentic AI applications. Tests prompt injections, handles vulnerability assessment, SBOM generation, and static analysis.
Custom licenseSep 10, 2026 · metrics 2.10.0
PyPI · npm
confident-ai/deepeval
The LLM Evaluation Framework
Apache-2.0Sep 16, 2026 · metrics 2.10.0
Go
praetorian-inc/julius
Simple LLM service identification - translate IP:Port to Ollama, vLLM, LiteLLM, or 60+ other AI services in seconds
Apache-2.0Sep 17, 2026 · metrics 2.10.0
PyPI
truera/trulens
Evaluation and Tracking for LLM Experiments and AI Agents
MITSep 17, 2026 · metrics 2.10.0
PyPI · npm
overmind-core/overmind
The platform for continuously improving AI agents.
AGPL-3.0Oct 19, 2026 · metrics 2.10.0
PyPI · npm
UiPath/coder_eval
Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.
Apache-2.0Oct 19, 2026 · metrics 2.10.0
PyPI
cvs-health/uqlm
[JMLR 2026] "UQLM: A Python Package for Uncertainty Quantification in Large Language Models"
Apache-2.0Sep 19, 2026 · metrics 2.10.0
PyPI
JudgmentLabs/judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
Apache-2.0Sep 17, 2026 · metrics 2.10.0
npm
microsoft/prompty
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
MITSep 17, 2026 · metrics 2.10.0
PyPI · npm
Marker-Inc-Korea/AutoRAG
AutoRAG: Now your agent can find anything in your computer. It gets smarter if you are using it frequently.
Custom licenseSep 15, 2026 · metrics 2.10.0
Go · Maven · npm
hugalafutro/model-hotel
Multi-Provider AI Gateway - No personal logs by design. Model autodiscovery, Failover groups, High availability, Android companion app, and more - "Because we have LiteLLM at home"
MITSep 10, 2026 · metrics 2.10.0
Hex · npm
ccarvalho-eng/aludel
Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir
Apache-2.0Sep 9, 2026 · metrics 2.10.0
PyPI · npm
alizahidraja/isnad
Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines
Apache-2.0Oct 21, 2026 · metrics 2.10.0
PyPI
b7n0de/proofbundle
Offline cryptographic receipts for AI evaluation results — Ed25519 + RFC 6962 Merkle + optional SD-JWT. Integrity, not truth
MITSep 13, 2026 · metrics 2.10.0
PyPI
Sahil170595/quantfit
Quantize an LLM and check it still refuses what it should — a GPU-aware quantization CLI that measures the safety drift of the quantization it just performed.
Apache-2.0Oct 12, 2026 · metrics 2.10.0
PyPI
scaleapi/agentenv-framework
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
Apache-2.0Oct 6, 2026 · metrics 2.10.0
npm
Precisa-Saude/medbench-brasil
Leaderboard reproduzível de LLMs em provas médicas brasileiras (Revalida, INEP) — zero-shot, 3 execuções, análise de contaminação
Custom licenseOct 18, 2026 · metrics 2.10.0
npm
huggingface/aisheets
Build, enrich, and transform datasets using AI models with no code
Apache-2.0Sep 19, 2026 · metrics 2.10.0
PyPI
phierceweb/pf-core
Python foundation for LLM apps whose prompts and spend you can actually see — versioned prompts, every call recorded and replayable, budgets, evals, jobs.
MITSep 5, 2026 · metrics 2.10.0
PyPI · npm
fastaifoundry/fastaiagent-sdk
No repository description published.
Apache-2.0Sep 10, 2026 · metrics 2.10.0
RubyGems
homemade-software-inc/completion-kit
Your prompts need tests too. Run prompts against real datasets, score outputs with LLM judges, version everything, and compare runs to see what got better.
Custom licenseSep 10, 2026 · metrics 2.10.0
PyPI
HarperZ9/flywheel
Flywheel runs AI tasks with a model you choose and keeps a receipt anyone can recheck offline.
Custom licenseOct 15, 2026 · metrics 2.10.0