Go
bzick/tokenizer
Tokenizer (lexer) for golang
MIT5. Sept. 2026 · Metriken 2.10.0
Alle Tags
Katalog-Tag
#tokenizer
Alle Repositories im öffentlichen Register, die dieses Tag tragen — aus ihren GitHub-Topics oder den von ihren Paket-Registries veröffentlichten Schlagwörtern. Die Gesundheit wird nach derselben versionierten Methodik gemessen wie im übrigen Register.
76 Einträge
npm
dazl-dev/tokey
Simple general purpose tokenizer
MIT15. Sept. 2026 · Metriken 2.10.0
Packagist
loupe-php/matcher
A library to highlight and crop terms and context within strings such as search results
MIT10. Sept. 2026 · Metriken 2.10.0
PyPI · crates.io
marcelroed/gigatoken
Language model tokenization at GB/s
MIT12. Sept. 2026 · Metriken 2.10.0
npm
webpro-nl/unbash
Fast 0-deps bash parser written in TypeScript
ISC27. Aug. 2026 · Metriken 2.10.0
NuGet
ProwlEngine/Anthology
The collection of libraries that power the Prowl game engine.
MIT13. Sept. 2026 · Metriken 2.10.0
crates.io · npm · PyPI
bio-rs/bio-rs
AI-ready biological data I/O, validation, and tokenization engine
Eigene Lizenz8. Sept. 2026 · Metriken 2.10.0
NuGet
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
MIT15. Sept. 2026 · Metriken 2.10.0
npm · Go
lancekrogers/tcount
Local Tokenizer/Counter
MIT11. Sept. 2026 · Metriken 2.10.0
crates.io
lindera/lindera-tantivy
Lindera tokenizer for Tantivy.
MIT17. Sept. 2026 · Metriken 2.10.0
Packagist
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Eigene Lizenz27. Aug. 2026 · Metriken 2.10.0
NuGet
umutozel/Jokenizer.Net
:hash: C# Expression parser and evaluator, inspired from jokenizer project.
MIT10. Sept. 2026 · Metriken 2.10.0
Go
polydawn/refmt
Object mapping for golang.
MIT23. Sept. 2026 · Metriken 2.10.0
Go
chadsr/ollamatokenizer
Ollama's internal tokenization as API endpoints.
MIT20. Okt. 2026 · Metriken 2.10.0
npm
Borewit/token-types
A primitive token library used to read from, and to write a node Buffer
MIT23. Sept. 2026 · Metriken 2.10.0
Go
clipperhouse/uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
MIT14. Sept. 2026 · Metriken 2.10.0
npm
csstree/csstree
A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations
MIT16. Sept. 2026 · Metriken 2.10.0
crates.io
linebender/svgtypes
A collection of parsers for SVG types.
Apache-2.019. Sept. 2026 · Metriken 2.10.0
Go
ron2111/omnitoken
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
MIT22. Okt. 2026 · Metriken 2.10.0
PyPI
tusharsadhwani/pytokens
A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
MIT9. Sept. 2026 · Metriken 2.10.0
crates.io
jorge-menjivar/tekken-rs
Rust implementation of the Mistral Tekken tokenizer
Apache-2.023. Aug. 2026 · Metriken 2.10.0
npm
lydell/js-tokens
Tiny JavaScript tokenizer.
MIT13. Sept. 2026 · Metriken 2.10.0
npm
Borewit/strtok3
Promise based streaming tokenizer
MIT23. Sept. 2026 · Metriken 2.10.0
npm
qwertymuzaffar/chunklet
Token-aware text chunking for RAG - exact source offsets, sentence and markdown modes, zero dependencies.
MIT20. Okt. 2026 · Metriken 2.10.0
npm
siliconvalleyglobal/TokenSaveOS
Keine Repository-Beschreibung veröffentlicht.
Eigene Lizenz18. Sept. 2026 · Metriken 2.10.0
Packagist
spiral/tokenizer
[READ ONLY] Locate available classes by parent, interface or trait. Subtree split of the Spiral Tokenizer component (see spiral/framework)
MIT20. Okt. 2026 · Metriken 2.10.0
PyPI
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MIT27. Aug. 2026 · Metriken 2.10.0
PyPI
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
MIT12. Sept. 2026 · Metriken 2.10.0
npm
Borewit/tokenizer-inflate
A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.
MIT16. Sept. 2026 · Metriken 2.10.0
PyPI
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Apache-2.012. Sept. 2026 · Metriken 2.10.0