All tags
Catalogue tag

#bpe

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

6 records
Tagged “bpe”Ranked by health index
crates.io
80Goodhealth index
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Rust · Python★ 10.9K↓ 2.9M/moJul 16, 2026
Apache-2.0Jul 16, 2026 · metrics 1.13.0
Go
58Moderatehealth index
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
Go★ 6Jul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
Go · npm
55Moderatehealth index
lancekrogers/tcount
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
Go★ 2↓ 19/moJul 13, 2026
MITJul 13, 2026 · metrics 1.13.0
Go · npm
54Moderatehealth index
lancekrogers/go-token-counter
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
Go★ 2Jul 18, 2026
MITJul 18, 2026 · metrics 1.13.0
PyPI · crates.io
50Moderatehealth index
marcelroed/gigatoken
Language model tokenization at GB/s
Rust · Python★ 115↓ 2,741/moJul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
npm
13Criticalhealth index
plurnk/plurnk-mimetypes-tokenizers
Bundled LLM tokenizer vocabularies for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math
JavaScript★ 0↓ 2,529/moJul 15, 2026
MITJul 15, 2026 · metrics 1.13.0