All tags
Catalogue tag

#tokenize

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

6 records
Tagged “tokenize”Ranked by health index
PyPI
77Goodhealth index
gristlabs/asttokens
Annotate Python AST trees with source text and token information
Python★ 192Jul 20, 2026
Apache-2.0Jul 20, 2026 · metrics 2.10.0
Packagist
65Goodhealth index
loupe-php/matcher
A library to highlight and crop terms and context within strings such as search results
PHP★ 10↓ 20.6K/moJul 17, 2026
MITJul 17, 2026 · metrics 2.10.0
npm
57Moderatehealth index
micromark/micromark
small, safe, and great commonmark (optionally gfm, mdx) compliant markdown parser
JavaScript★ 2,199↓ 1.4B/moAug 4, 2026
MITAug 4, 2026 · metrics 2.10.0
PyPI
47Weakhealth index
OpenVoiceOS/quebra_frases
chunks strings into byte sized pieces
Python★ 1Jul 21, 2026
Apache-2.0Jul 21, 2026 · metrics 2.10.0
crates.io
34At Riskhealth index
stevedonovan/scanlex
A simple lexical scanner in Rust
Rust★ 12↓ 379.4K/moJul 25, 2026
MITJul 25, 2026 · metrics 2.10.0