Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MITAug 23, 2026 · metrics 2.10.0
All tags
Catalogue tag
#tokenizer
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
54 records
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MITSep 5, 2026 · metrics 2.10.0
Go · npm
lancekrogers/go-token-counter
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
MITJul 18, 2026 · metrics 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MITAug 22, 2026 · metrics 2.10.0
—
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MITJul 17, 2026 · metrics 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MITJul 22, 2026 · metrics 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MITJul 17, 2026 · metrics 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.0Aug 23, 2026 · metrics 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MITJul 22, 2026 · metrics 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MITAug 27, 2026 · metrics 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.0Jul 21, 2026 · metrics 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MITAug 4, 2026 · metrics 2.10.0
crates.io
RazrFalcon/xmlparser
A low-level, pull-based, zero-allocation XML 1.0 parser.
Apache-2.0Aug 13, 2026 · metrics 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MITJul 21, 2026 · metrics 2.10.0
crates.io
savannstm/language-tokenizer
Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
WTFPLAug 20, 2026 · metrics 2.10.0
Packagist
sebastianbergmann/php-token-stream
Wrapper around PHP's tokenizer extension.
Custom licenseAug 27, 2026 · metrics 2.10.0
Packagist
phplrt/lexer
[READ ONLY] Subtree split of the phplrt lexer component (see phplrt/phplrt)
MITJul 24, 2026 · metrics 2.10.0
Packagist
phplrt/lexer-contracts
[READ ONLY] Subtree split of the phplrt lexer contracts component (see phplrt/phplrt)
MITJul 24, 2026 · metrics 2.10.0
—
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Custom licenseJul 22, 2026 · metrics 2.10.0
crates.io
fschutt/simplecss
A very simple CSS 2.1 tokenizer
MPL-2.0Aug 23, 2026 · metrics 2.10.0
npm
fent/ret.js
Tokenizes a string that represents a regular expression.
MITAug 31, 2026 · metrics 2.10.0
Packagist
nette/tokenizer
[DISCONTINUED] Source code tokenizer
Custom licenseAug 16, 2026 · metrics 2.10.0
npm
plurnk/plurnk-providers
Framework + contract for @plurnk/plurnk-providers-* sibling packages (LLM transports + tokenizer + cost accounting).
MITJul 15, 2026 · metrics 2.10.0
npm
plurnk/plurnk-mimetypes-tokenizers
Bundled LLM tokenizer vocabularies for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math
MITJul 15, 2026 · metrics 2.10.0