#tokenizer

该仓库未发布描述。

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

Graph-based tool retrieval for LLM agents — 248 tools → 82% accuracy, 79% fewer tokens. Zero dependencies. OpenAPI / MCP / LangChain.

Scanner/lexer with sufficient regex support and multiple scanner modes

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

An Erlang source code tokenizer written in Rust.

A pure-Rust graphics engine for PDF, PostScript, and EPS.

Tokenizes a string that represents a regular expression.

JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification

초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

High-performance Rust tiktoken BPE tokenizer + WASM bindings (@goliapkg/tiktoken-wasm)

A PSR-2-R code sniffer and code-style auto-correction-tool - including many useful additions

High-performance CSS parser optimized for static analysis and formatting

Kura — the knowledgebase for humans and agents. Agent-native docs infrastructure, built on June.