Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MIT23 ago 2026 · métricas 2.10.0
Todas las etiquetas
Etiqueta del catálogo
#tokenizer
Todos los repositorios del registro público que llevan esta etiqueta, procedente de sus topics de GitHub o de las palabras clave que publican sus registros de paquetes. La salud se mide con la misma metodología versionada que el resto del registro.
54 registros
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MIT5 sept 2026 · métricas 2.10.0
Go · npm
lancekrogers/go-token-counter
Local-first LLM token counter: exact OpenAI BPE, Claude/Gemini estimates, context-window usage. Zero network. CLI + Go library.
MIT18 jul 2026 · métricas 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MIT22 ago 2026 · métricas 2.10.0
—
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MIT17 jul 2026 · métricas 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MIT22 jul 2026 · métricas 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MIT17 jul 2026 · métricas 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.023 ago 2026 · métricas 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MIT22 jul 2026 · métricas 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MIT27 ago 2026 · métricas 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.021 jul 2026 · métricas 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MIT4 ago 2026 · métricas 2.10.0
crates.io
RazrFalcon/xmlparser
A low-level, pull-based, zero-allocation XML 1.0 parser.
Apache-2.013 ago 2026 · métricas 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MIT21 jul 2026 · métricas 2.10.0
crates.io
savannstm/language-tokenizer
Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
WTFPL20 ago 2026 · métricas 2.10.0
Packagist
sebastianbergmann/php-token-stream
Wrapper around PHP's tokenizer extension.
Licencia propia27 ago 2026 · métricas 2.10.0
Packagist
phplrt/lexer
[READ ONLY] Subtree split of the phplrt lexer component (see phplrt/phplrt)
MIT24 jul 2026 · métricas 2.10.0
Packagist
phplrt/lexer-contracts
[READ ONLY] Subtree split of the phplrt lexer contracts component (see phplrt/phplrt)
MIT24 jul 2026 · métricas 2.10.0
—
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Licencia propia22 jul 2026 · métricas 2.10.0
crates.io
fschutt/simplecss
A very simple CSS 2.1 tokenizer
MPL-2.023 ago 2026 · métricas 2.10.0
npm
fent/ret.js
Tokenizes a string that represents a regular expression.
MIT31 ago 2026 · métricas 2.10.0
Packagist
nette/tokenizer
[DISCONTINUED] Source code tokenizer
Licencia propia16 ago 2026 · métricas 2.10.0
npm
plurnk/plurnk-providers
Framework + contract for @plurnk/plurnk-providers-* sibling packages (LLM transports + tokenizer + cost accounting).
MIT15 jul 2026 · métricas 2.10.0
npm
plurnk/plurnk-mimetypes-tokenizers
Bundled LLM tokenizer vocabularies for @plurnk/plurnk-mimetypes' tokenizer seam — exact token counting for window math
MIT15 jul 2026 · métricas 2.10.0