All tags
Catalogue tag

#tokenizer

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

27 records
Tagged “tokenizer”Ranked by health index
crates.io
80Goodhealth index
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Rust · Python★ 10.9K↓ 2.9M/moJul 16, 2026
Apache-2.0Jul 16, 2026 · metrics 1.13.0
npm
74Goodhealth index
csstools/postcss-plugins
PostCSS Tools and Plugins
CSS · JavaScript · TypeScript★ 1,051↓ 906M/moJul 22, 2026
MIT-0Jul 22, 2026 · metrics 1.13.0
crates.io · npm · Packagist +2
72Goodhealth index
lindera/lindera
A multilingual morphological analysis library.
Rust★ 644↓ 481.1K/moJul 18, 2026
MITJul 18, 2026 · metrics 1.13.0
Packagist
61Moderatehealth index
loupe-php/matcher
A library to highlight and crop terms and context within strings such as search results
PHP★ 10↓ 20.6K/moJul 17, 2026
MITJul 17, 2026 · metrics 1.13.0
Go
58Moderatehealth index
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
Go★ 6Jul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
Go
57Moderatehealth index
bzick/tokenizer
Tokenizer (lexer) for golang
Go★ 141Jul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
npm
57Moderatehealth index
csstree/csstree
A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations
JavaScript★ 2,111↓ 399M/moJul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
crates.io
56Moderatehealth index
sile/erl_tokenize
An Erlang source code tokenizer written in Rust.
Rust★ 12↓ 2,179/moJul 22, 2026
Apache-2.0Jul 22, 2026 · metrics 1.13.0
npm
56Moderatehealth index
webpro-nl/unbash
Fast 0-deps bash parser written in TypeScript
Shell · TypeScript★ 34↓ 29.9M/moJul 17, 2026
ISCJul 17, 2026 · metrics 1.13.0
PyPI
55Moderatehealth index
bitextor/bitextor
Bitextor generates translation memories from multilingual websites
Python · Shell★ 299Jul 21, 2026
GPL-3.0Jul 21, 2026 · metrics 1.13.0
Go · npm
55Moderatehealth index
lancekrogers/tcount
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
Go★ 2↓ 19/moJul 13, 2026
MITJul 13, 2026 · metrics 1.13.0
—
55Moderatehealth index
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
C#★ 11Jul 17, 2026
MITJul 17, 2026 · metrics 1.13.0
Go
54Moderatehealth index
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
Go★ 1↓ 0/moJul 14, 2026
MITJul 14, 2026 · metrics 1.13.0
Go · npm
54Moderatehealth index
lancekrogers/go-token-counter
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
Go★ 2Jul 18, 2026
MITJul 18, 2026 · metrics 1.13.0
Packagist
54Moderatehealth index
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
PHP★ 18↓ 425.3K/moJul 13, 2026
MITJul 13, 2026 · metrics 1.13.0
PyPI
53Moderatehealth index
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
Python★ 4Jul 17, 2026
MITJul 17, 2026 · metrics 1.13.0
npm
52Moderatehealth index
lydell/js-tokens
Tiny JavaScript tokenizer.
JavaScript · CoffeeScript★ 556Jul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
PyPI
50Moderatehealth index
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
Python★ 497↓ 2.4M/moJul 21, 2026
MITJul 21, 2026 · metrics 1.13.0
PyPI · crates.io
50Moderatehealth index
marcelroed/gigatoken
Language model tokenization at GB/s
Rust · Python★ 115↓ 2,741/moJul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
PyPI
48At riskhealth index
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Python★ 1Jul 21, 2026
Apache-2.0Jul 21, 2026 · metrics 1.13.0
npm
47At riskhealth index
borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
TypeScript★ 0↓ 83.2M/moJul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
PyPI
46At riskhealth index
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
C++ · Cython · Python★ 533Jul 21, 2026
MITJul 21, 2026 · metrics 1.13.0
Packagist
44At riskhealth index
mck89/peast
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
PHP★ 192↓ 1.4M/moJul 21, 2026
BSD-3-ClauseJul 21, 2026 · metrics 1.13.0
—
30At riskhealth index
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Julia★ 104Jul 22, 2026
Custom licenseJul 22, 2026 · metrics 1.13.0
Packagist
28Criticalhealth index
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHP★ 5,199↓ 14.3M/moJul 20, 2026
Custom licenseJul 20, 2026 · metrics 1.13.0
npm
16Criticalhealth index
plurnk/plurnk-providers
Framework + contract for @plurnk/plurnk-providers-* sibling packages (LLM transports + tokenizer + cost accounting).
TypeScript · JavaScript★ 0↓ 9,778/moJul 15, 2026
MITJul 15, 2026 · metrics 1.13.0
npm
13Criticalhealth index
plurnk/plurnk-mimetypes-tokenizers
Bundled LLM tokenizer vocabularies for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math
JavaScript★ 0↓ 2,529/moJul 15, 2026
MITJul 15, 2026 · metrics 1.13.0