crates.io
RazrFalcon/xmlparser
A low-level, pull-based, zero-allocation XML 1.0 parser.
Apache-2.0Sep 18, 2026 · metrics 2.10.0
All tags
Catalogue tag
#tokenizer
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
76 records
npm · Maven · RubyGems
leonsilicon/react-native-jieba
Turbo Module integrating cppjieba for React Native.
MITOct 5, 2026 · metrics 2.10.0
crates.io
savannstm/language-tokenizer
Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
WTFPLOct 19, 2026 · metrics 2.10.0
npm
ericprud/ts-jison
Bison in JavaScript.
MITOct 7, 2026 · metrics 2.10.0
Packagist
sebastianbergmann/php-token-stream
Wrapper around PHP's tokenizer extension.
Custom licenseAug 27, 2026 · metrics 2.10.0
PyPI
bitextor/bitextor
Bitextor generates translation memories from multilingual websites
GPL-3.0Sep 12, 2026 · metrics 2.10.0
Packagist
phplrt/lexer
[READ ONLY] Subtree split of the phplrt lexer component (see phplrt/phplrt)
MITSep 13, 2026 · metrics 2.10.0
Packagist
phplrt/lexer-builder
[READ ONLY] Subtree split of the phplrt lexer component (see phplrt/phplrt)
MITSep 15, 2026 · metrics 2.10.0
Packagist
phplrt/lexer-contracts
[READ ONLY] Subtree split of the phplrt lexer contracts component (see phplrt/phplrt)
MITSep 13, 2026 · metrics 2.10.0
—
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Custom licenseSep 12, 2026 · metrics 2.10.0
crates.io
fschutt/simplecss
A very simple CSS 2.1 tokenizer
MPL-2.0Aug 23, 2026 · metrics 2.10.0
crates.io
spider-rs/markup5ever
Send-able fork of markup5ever, depends on spider-tendril
Apache-2.0Sep 15, 2026 · metrics 2.10.0
npm
Borewit/tokenizer-token
TypeScript definition for strtok3 token
No licenseSep 23, 2026 · metrics 2.10.0
Packagist
chriskonnertz/string-calc
PHP calculator library for mathematical terms (expressions) passed as strings
MITOct 18, 2026 · metrics 2.10.0
Packagist
nette/tokenizer
[DISCONTINUED] Source code tokenizer
Custom licenseSep 18, 2026 · metrics 2.10.0
npm
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD-3-ClauseSep 19, 2026 · metrics 2.10.0