Go
bzick/tokenizer
Tokenizer (lexer) for golang
MITSep 5, 2026 · metrics 2.10.0
All tags
Catalogue tag
#tokenizer
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
76 records
npm
dazl-dev/tokey
Simple general purpose tokenizer
MITSep 15, 2026 · metrics 2.10.0
Packagist
loupe-php/matcher
A library to highlight and crop terms and context within strings such as search results
MITSep 10, 2026 · metrics 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MITSep 12, 2026 · metrics 2.10.0
npm
webpro-nl/unbash
Fast 0-deps bash parser written in TypeScript
ISCAug 27, 2026 · metrics 2.10.0
NuGet
ProwlEngine/Anthology
The collection of libraries that power the Prowl game engine.
MITSep 13, 2026 · metrics 2.10.0
crates.io · npm · PyPI
bio-rs/bio-rs
AI-ready biological data I/O, validation, and tokenization engine
Custom licenseSep 8, 2026 · metrics 2.10.0
NuGet
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
MITSep 15, 2026 · metrics 2.10.0
npm · Go
lancekrogers/tcount
Local Tokenizer/Counter
MITSep 11, 2026 · metrics 2.10.0
crates.io
lindera/lindera-tantivy
Lindera tokenizer for Tantivy.
MITSep 17, 2026 · metrics 2.10.0
Packagist
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Custom licenseAug 27, 2026 · metrics 2.10.0
NuGet
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MITSep 10, 2026 · metrics 2.10.0
Go
polydawn/refmt
Object mapping for golang.
MITSep 23, 2026 · metrics 2.10.0
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MITSep 5, 2026 · metrics 2.10.0
Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MITOct 20, 2026 · metrics 2.10.0
npm
Borewit/token-types
A primitive token library used to read from, and to write a node Buffer
MITSep 23, 2026 · metrics 2.10.0
Go
clipperhouse/uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
MITSep 14, 2026 · metrics 2.10.0
npm
csstree/csstree
A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations
MITSep 16, 2026 · metrics 2.10.0
crates.io
linebender/svgtypes
A collection of parsers for SVG types.
Apache-2.0Sep 19, 2026 · metrics 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MITSep 9, 2026 · metrics 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.0Aug 23, 2026 · metrics 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MITSep 13, 2026 · metrics 2.10.0
npm
Borewit/strtok3
Promise based streaming tokenizer
MITSep 23, 2026 · metrics 2.10.0
npm
qwertymuzaffar/chunklet
Token-aware text chunking for RAG - exact source offsets, sentence and markdown modes, zero dependencies.
MITOct 20, 2026 · metrics 2.10.0
npm
siliconvalleyglobal/TokenSaveOS
No repository description published.
Custom licenseSep 18, 2026 · metrics 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MITOct 20, 2026 · metrics 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MITAug 27, 2026 · metrics 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MITSep 12, 2026 · metrics 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MITSep 16, 2026 · metrics 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.0Sep 12, 2026 · metrics 2.10.0