#tokenizer

chunks strings into byte sized pieces

A package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification

Tokenization for Julia source code