All tags
Catalogue tag

#tokenization

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

7 records
Tagged “tokenization”Ranked by health index
npm
71Goodhealth index
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.
TypeScript★ 24.9KJul 19, 2026
MITJul 19, 2026 · metrics 1.13.0
PyPI · npm
69Moderatehealth index
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
Python · MDX · Cython★ 33.7K↓ 22M/moJul 17, 2026
MITJul 17, 2026 · metrics 1.13.0
Go
58Moderatehealth index
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
Go★ 6Jul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
Go
54Moderatehealth index
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
Go★ 1↓ 0/moJul 14, 2026
MITJul 14, 2026 · metrics 1.13.0
crates.io
53Moderatehealth index
bio-rs/bio-rs
AI-ready biological data I/O, validation, and tokenization engine
Rust · Python★ 3Jul 15, 2026
Custom licenseJul 15, 2026 · metrics 1.13.0
PyPI · crates.io
50Moderatehealth index
marcelroed/gigatoken
Language model tokenization at GB/s
Rust · Python★ 115↓ 2,741/moJul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
PyPI
48At riskhealth index
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Python★ 1Jul 21, 2026
Apache-2.0Jul 21, 2026 · metrics 1.13.0