Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MIT23. Aug. 2026 · Metriken 2.10.0
Alle Tags
Katalog-Tag
#tokenizer
Alle Repositories im öffentlichen Register, die dieses Tag tragen — aus ihren GitHub-Topics oder den von ihren Paket-Registries veröffentlichten Schlagwörtern. Die Gesundheit wird nach derselben versionierten Methodik gemessen wie im übrigen Register.
54 Einträge
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MIT5. Sept. 2026 · Metriken 2.10.0
Go · npm
lancekrogers/go-token-counter
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
MIT18. Juli 2026 · Metriken 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MIT22. Aug. 2026 · Metriken 2.10.0
—
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MIT17. Juli 2026 · Metriken 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MIT22. Juli 2026 · Metriken 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MIT17. Juli 2026 · Metriken 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.023. Aug. 2026 · Metriken 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MIT22. Juli 2026 · Metriken 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MIT27. Aug. 2026 · Metriken 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.021. Juli 2026 · Metriken 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MIT4. Aug. 2026 · Metriken 2.10.0
crates.io
RazrFalcon/xmlparser
A low-level, pull-based, zero-allocation XML 1.0 parser.
Apache-2.013. Aug. 2026 · Metriken 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MIT21. Juli 2026 · Metriken 2.10.0
crates.io
savannstm/language-tokenizer
Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
WTFPL20. Aug. 2026 · Metriken 2.10.0
Packagist
sebastianbergmann/php-token-stream
Wrapper around PHP's tokenizer extension.
Eigene Lizenz27. Aug. 2026 · Metriken 2.10.0
Packagist
phplrt/lexer
[READ ONLY] Subtree split of the phplrt lexer component (see phplrt/phplrt)
MIT24. Juli 2026 · Metriken 2.10.0
Packagist
phplrt/lexer-contracts
[READ ONLY] Subtree split of the phplrt lexer contracts component (see phplrt/phplrt)
MIT24. Juli 2026 · Metriken 2.10.0
—
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Eigene Lizenz22. Juli 2026 · Metriken 2.10.0
crates.io
fschutt/simplecss
A very simple CSS 2.1 tokenizer
MPL-2.023. Aug. 2026 · Metriken 2.10.0
npm
fent/ret.js
Tokenizes a string that represents a regular expression.
MIT31. Aug. 2026 · Metriken 2.10.0
Packagist
nette/tokenizer
[DISCONTINUED] Source code tokenizer
Eigene Lizenz16. Aug. 2026 · Metriken 2.10.0
npm
plurnk/plurnk-providers
Framework + contract for @plurnk/plurnk-providers-* sibling packages (LLM transports + tokenizer + cost accounting).
MIT15. Juli 2026 · Metriken 2.10.0
npm
plurnk/plurnk-mimetypes-tokenizers
Bundled LLM tokenizer vocabularies for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math
MIT15. Juli 2026 · Metriken 2.10.0