Go
bzick/tokenizer
Tokenizer (lexer) for golang
MIT5 sept 2026 · métricas 2.10.0
Todas las etiquetas
Etiqueta del catálogo
#tokenizer
Todos los repositorios del registro público que llevan esta etiqueta, procedente de sus topics de GitHub o de las palabras clave que publican sus registros de paquetes. La salud se mide con la misma metodología versionada que el resto del registro.
76 registros
npm
dazl-dev/tokey
Simple general purpose tokenizer
MIT15 sept 2026 · métricas 2.10.0
Packagist
loupe-php/matcher
A library to highlight and crop terms and context within strings such as search results
MIT10 sept 2026 · métricas 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MIT12 sept 2026 · métricas 2.10.0
npm
webpro-nl/unbash
Fast 0-deps bash parser written in TypeScript
ISC27 ago 2026 · métricas 2.10.0
NuGet
ProwlEngine/Anthology
The collection of libraries that power the Prowl game engine.
MIT13 sept 2026 · métricas 2.10.0
crates.io · npm · PyPI
bio-rs/bio-rs
AI-ready biological data I/O, validation, and tokenization engine
Licencia propia8 sept 2026 · métricas 2.10.0
NuGet
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
MIT15 sept 2026 · métricas 2.10.0
npm · Go
lancekrogers/tcount
Local Tokenizer/Counter
MIT11 sept 2026 · métricas 2.10.0
crates.io
lindera/lindera-tantivy
Lindera tokenizer for Tantivy.
MIT17 sept 2026 · métricas 2.10.0
Packagist
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Licencia propia27 ago 2026 · métricas 2.10.0
NuGet
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MIT10 sept 2026 · métricas 2.10.0
Go
polydawn/refmt
Object mapping for golang.
MIT23 sept 2026 · métricas 2.10.0
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MIT5 sept 2026 · métricas 2.10.0
Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MIT20 oct 2026 · métricas 2.10.0
npm
Borewit/token-types
A primitive token library used to read from, and to write a node Buffer
MIT23 sept 2026 · métricas 2.10.0
Go
clipperhouse/uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
MIT14 sept 2026 · métricas 2.10.0
npm
csstree/csstree
A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations
MIT16 sept 2026 · métricas 2.10.0
crates.io
linebender/svgtypes
A collection of parsers for SVG types.
Apache-2.019 sept 2026 · métricas 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MIT9 sept 2026 · métricas 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.023 ago 2026 · métricas 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MIT13 sept 2026 · métricas 2.10.0
npm
Borewit/strtok3
Promise based streaming tokenizer
MIT23 sept 2026 · métricas 2.10.0
npm
qwertymuzaffar/chunklet
Token-aware text chunking for RAG - exact source offsets, sentence and markdown modes, zero dependencies.
MIT20 oct 2026 · métricas 2.10.0
npm
siliconvalleyglobal/TokenSaveOS
El repositorio no publica descripción.
Licencia propia18 sept 2026 · métricas 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MIT20 oct 2026 · métricas 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MIT27 ago 2026 · métricas 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MIT12 sept 2026 · métricas 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MIT16 sept 2026 · métricas 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.012 sept 2026 · métricas 2.10.0