feyninc / tokie

Cuándo se añadió cada estrella y fork, recopilado de GitHub y agrupado por día. El crecimiento acumulado se sitúa justo encima de las adiciones diarias que lo componen, de modo que ambos se leen en conjunto: la acumulación orgánica sostenida no se parece en nada a un pico abrupto y efímero. Cuando esa diferencia es medible, se informa como autenticidad del crecimiento.

Conjunto completo de dependencias resueltas según el grafo de dependencias de GitHub: 48 paquetes directos y 259 indirectos (transitivos). El cierre transitivo es completo cuando el repositorio incluye un lockfile.

Este repositorio no publica ningún paquete que el índice resuelva, así que se evaluó su propio grafo de dependencias — 281 paquetes, que incluyen también fijaciones de desarrollo y prueba que nunca se distribuyen: 2 tienen avisos conocidos, de los cuales 0 son directas. 26 no pudieron evaluarse: sin versión resuelta, ecosistema no admitido, o fuera de la lista de paquetes informada.

Un aviso significa que la versión registrada en el grafo de dependencias cae dentro del rango afectado de un aviso. No se analiza la alcanzabilidad, y el grafo incluye fijaciones de desarrollo y prueba: un hallazgo puede referirse al utillaje y no al software distribuido.

{
  "data": {
    "repo": {
      "topics": [],
      "is_fork": false,
      "size_kb": 3713,
      "has_wiki": true,
      "homepage": null,
      "languages": {
        "Rust": 1051481,
        "Shell": 1514,
        "Python": 78768
      },
      "pushed_at": "2026-07-24T09:50:59Z",
      "created_at": "2025-12-18T21:15:08Z",
      "owner_type": "Organization",
      "updated_at": "2026-07-24T09:52:27Z",
      "description": "🍡 30x faster tokenization for every HuggingFace model",
      "is_archived": false,
      "is_disabled": false,
      "license_spdx": "Apache-2.0",
      "default_branch": "main",
      "license_spdx_raw": "Apache-2.0",
      "primary_language": "Rust",
      "significant_languages": [
        "Rust"
      ]
    },
    "owner": {
      "blog": "https://usefeyn.com",
      "name": "Feyn",
      "type": "Organization",
      "login": "feyninc",
      "company": null,
      "location": "United States of America",
      "followers": 150,
      "avatar_url": "https://avatars.githubusercontent.com/u/205278415?v=4",
      "created_at": "2025-03-28T06:41:10Z",
      "is_verified": null,
      "public_repos": 16,
      "account_age_days": 483
    },
    "license": {
      "state": "standard",
      "spdx_id": "Apache-2.0",
      "raw_spdx": "Apache-2.0",
      "file_present": true,
      "scorecard_found": true,
      "profile_has_license": true
    },
    "activity": {
      "releases": [
        {
          "tag": "v0.1.3",
          "kind": "patch",
          "published_at": "2026-07-24T09:38:18Z"
        },
        {
          "tag": "v0.1.2",
          "kind": "patch",
          "published_at": "2026-07-24T04:41:22Z"
        },
        {
          "tag": "v0.1.1",
          "kind": "patch",
          "published_at": "2026-07-23T22:09:58Z"
        },
        {
          "tag": "v0.1.0",
          "kind": "minor",
          "published_at": "2026-07-23T21:51:08Z"
        },
        {
          "tag": "v0.0.10",
          "kind": "patch",
          "published_at": "2026-05-28T17:51:58Z"
        },
        {
          "tag": "0.0.10-alpha.2",
          "kind": "prerelease",
          "published_at": "2026-05-28T17:20:19Z"
        },
        {
          "tag": "v0.0.10-alpha.1",
          "kind": "prerelease",
          "published_at": "2026-05-28T17:11:08Z"
        },
        {
          "tag": "v0.0.9",
          "kind": "patch",
          "published_at": "2026-05-05T18:17:29Z"
        },
        {
          "tag": "v0.0.8",
          "kind": "patch",
          "published_at": "2026-03-30T07:38:17Z"
        },
        {
          "tag": "v0.0.7",
          "kind": "patch",
          "published_at": "2026-03-26T02:43:10Z"
        },
        {
          "tag": "v0.0.6",
          "kind": "patch",
          "published_at": "2026-03-26T02:11:31Z"
        },
        {
          "tag": "v0.0.5",
          "kind": "patch",
          "published_at": "2026-03-24T02:42:14Z"
        },
        {
          "tag": "v0.0.4",
          "kind": "patch",
          "published_at": "2026-03-24T01:42:54Z"
        },
        {
          "tag": "v0.0.3",
          "kind": "patch",
          "published_at": "2026-03-23T06:39:36Z"
        }
      ],
      "recent_commits": [
        {
          "oid": "180d127346cf530c18b9229dfcdf4165bfe84e1f",
          "body": "The 0.1.3 crates.io publish failed with a 403 (token lacked publish\npermission) but the release still showed green: both publish steps had\ncontinue-on-error. Replace the long-lived CARGO_REGISTRY_TOKEN with\ncrates.io trusted publishing (OIDC via rust-lang/crates-io-auth-action,\nid-token: write) and \n[…]\n retries to ride out pretokie index\npropagation. Add workflow_dispatch (wheel/PyPI jobs gated to the release\nevent) so the crates half can be re-published without rebuilding wheels\nor re-hitting PyPI.",
          "is_bot": false,
          "headline": "ci: crates.io trusted publishing + fail loudly on publish errors",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T09:50:57Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "72e18c9739eb8323563ede7c55b404d17c525fbc",
          "body": "- New encode_files / count_tokens_files bulk file-encoding API\n- Cache-first encoder loop, rank-table merge core, fused pretokenize+encode\n- Mask-scanner bulk drain (pretokie 0.1.0 -> 0.1.3)\n- Fix: rank table silently disabled on .tkz load\n- pyo3/numpy 0.29 security bump",
          "is_bot": false,
          "headline": "Bump version to v0.1.3",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T09:37:54Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "436f7bd9d6d926c2018cf87c954eb4fb5a470677",
          "body": null,
          "is_bot": false,
          "headline": "Merge branch 'perf/pretok-throughput'",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T09:23:34Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "0a3c6391c4443263012fe1abb4435a911af88182",
          "body": "Phase 1 (profile_stages, gpt2/OWT 40MB, M3, medians of 7):\n  per-64B-batch: (a) NEON classify+movemask 7.4-7.8 ns; (b) boundary\n  algebra a+b 23.6-24.5 ns (b alone ~16.5); (c+d) full Mask walker\n  ~47-48 ns. Per piece (4.82 B/piece): Mask iterator 3.58-3.63,\n  Core scalar 7.5, scalar_advance isolate\n[…]\ne == Core); tokie --lib 154 green; 40k-iter\n  fuzz_differential green; owt_compare gpt2 20342/20342 and\n  DeepSeek-V3 20342/20342; piece_diff (new) 15MB gpt2 + deepseek,\n  iter 0 diffs / bulk 0 diffs.",
          "is_bot": false,
          "headline": "pretok: bulk-drain for_each_piece for the mask scanner",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T09:13:34Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "47b31deaa2d9d464c3098e3670fc61b57c169393",
          "body": "Fixes GHSA-36hh-v3qg-5jq4 (high, OOB read in PyList/PyTuple nth\niterators) and GHSA-chgr-c6px-7xpp (moderate, missing Sync bound on\nPyCFunction::new_closure), both patched in pyo3 0.29.0. Only\ntokie-python depends on pyo3/numpy; the tokie and pretokie crates are\nunaffected.\n\nMigrate the bindings for\n[…]\n::with_gil -> Python::attach, PyObject -> Py<PyAny>, and opt the\nEncoding pyclass out of the now-deprecated automatic FromPyObject\nderive with skip_from_py_object (it is only ever returned to Python).",
          "is_bot": false,
          "headline": "Bump pyo3 and numpy to 0.29 to fix Dependabot advisories",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T09:00:41Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "481d49c81af2cb164c414cf67915315b24ddf897",
          "body": "Phase 3: performance. On the 191MB OWT/gpt2 harness this takes\nencode_files from ~630 MB/s to ~880-990 MB/s fresh-process\n(~990-1030 MB/s warm); gigatoken-files interleaved on the same runs\nmeasures ~1040-1130 MB/s.\n\n- mmap large corpus files (libc, unix; read fallback elsewhere) with a\n  parallel p\n[…]\nge breakdown and the\n  fresh-process harness contract.\n\nGates: tokie lib 154 (141 baseline + 13 new), pretokie 69,\nowt_compare gpt2 20342/20342, Python identity vs encode_batch_flat on\n20MB OWT exact.",
          "is_bot": false,
          "headline": "Optimize the byte-source file pipeline to near-gigatoken throughput",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:46:47Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "75c984f587ae08b06bd6492a966a3c4e6c697600",
          "body": "Phase 2: Python surface. encode_files(paths, separator=b\"<|endoftext|>\",\nadd_special_tokens=False) returns (uint32 ids, uint64 offsets) numpy\narrays with zero-copy handoff; count_tokens_files returns the total.\nFile reading, document splitting, and encoding all happen in Rust with\nthe GIL released — no text crosses the Python boundary. Stubs updated.",
          "is_bot": false,
          "headline": "Expose encode_files and count_tokens_files in the Python binding",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:22:53Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "93ff264b863d7967dfaca53023205be6e8121609",
          "body": "Phase 1: Rust core. Read corpus files in Rust, split documents on a\nseparator byte sequence with memmem, drop empty documents (Python\nif-d-filter semantics), borrow &str views into the read buffer after one\nUTF-8 validation pass (lossy per-document fallback for invalid bytes),\nand encode with the existing steal_batches fused bulk pipeline. Returns\nflat ids plus ndocs+1 document offset boundaries.",
          "is_bot": false,
          "headline": "Add encode_files_flat and count_tokens_files byte-source bulk API",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:20:19Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "94505e1038e8e0344bb806edc2fc9e785761772b",
          "body": null,
          "is_bot": false,
          "headline": "Merge branch 'perf/fused-pipeline'",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:10:45Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "9f692e49f3f813050603d99f5aadf9a9baca10da",
          "body": "rebuild_pair_lookup inserted degenerate (id,id)->id entries for\nSplit::base tokens past the base range (added/special tokens such as\ngpt2's <|endoftext|>). RankPairTable::build correctly rejects such an\nentry via its merge-monotonicity check, which silently disabled the\nrank-merge miss path on every\n[…]\n pick up the fix on next load.\nVerified: has_rank_merge now true on the hub gpt2 path; lib 138/138;\nowt_compare 20342/20342 on gpt2 and DeepSeek-V3 (first run with\nrank-merge active on the .tkz path).",
          "is_bot": false,
          "headline": "fix: preserve rank-merge table across .tkz loads",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:10:22Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "d29da353ee9925a72d2ce2272b2947d66a5ffc4a",
          "body": "…cliffs\n\nTwo changes driven by the phase-1 numbers (see profile_glue_costs):\n\n1. Fused sequential loop. Pretokenizer::for_each_piece dispatches on the\n   pretokenizer type once per call and runs a monomorphized loop over the\n   concrete mask-scanner iterator; encode_sequential hoists the Encoder\n   \n[…]\nunt_tokens_batch/encode_batch_flat.\ncargo test -p tokie --lib 140 passed; -p pretokie --lib 69 passed;\nowt_compare 20342/20342 for gpt2 and DeepSeek-V3; rank_merge_diff\nIDENTITY HOLDS (0 divergences).",
          "is_bot": false,
          "headline": "perf: monomorphized fused pretokenize+encode loop; fix parallel-path …",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:04:04Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "eb1d9017cd581aa39a8576476faa9e87dae6fa47",
          "body": "profile_glue_costs decomposes the gap between component estimates\n(~4 ns/piece pretokenize + ~5.8 ns/piece hit-path encode) and the\nmeasured fused loop (~17 ns/piece). Measured on M3, gpt2/OWT 10MB,\nbest of 5, warm 2MiB cache:\n\n  A  pretok only, monomorphized     :  3.5 ns/piece  1363 MB/s\n  B  pret\n[…]\nhed. The 10-50KB bucket\n  (29% of OWT bytes) ran 140.7 MB/s vs 267.3 MB/s cached-sequential.\n\nDeepSeek shows the same shape plus a real added-token scan cost\n(4.1 ns/piece at 1.2 GB/s over all bytes).",
          "is_bot": false,
          "headline": "profile: phase-1 glue-cost decomposition for the ST bulk pipeline",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T08:03:46Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "d51e4de295394d8186cb4604c7b4e2ac4f619bb4",
          "body": "Combine the cache-first encoder loop with the rank-table merge core.\nThe pretoken-cache hit path keeps the two-u64 masked-load key, fixed\n3-slot emit, and outlined miss handling from perf/cache-first-loop.\nThe miss path (and the no-cache short-piece path) now dispatches to\nencode_rank_merge for piec\n[…]\n pretokie 69/69, rank_merge_diff\ndifferential identity 0 divergences (gpt2 + DeepSeek-V3, OWT pieces +\n400k fuzz strings), owt_compare 20342/20342 on both gpt2 and\nDeepSeek-V3, profile_hotpath builds.",
          "is_bot": false,
          "headline": "Merge perf/rank-merge-core into perf/cache-first-loop",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T07:37:48Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "c9b33b0b35265b5be1e6620e7003e7ca5a80e4fd",
          "body": "Phase-1 cycle accounting (new examples/profile_piece_costs.rs, gpt2 over\n10MB OWT, 2.07M pieces, M3, warm 2MiB cache) showed where the ~30ns/piece\nencode step actually went — NOT memory latency:\n\n  key construction (variable-len memcpy)   6.9 ns/hit\n  table probe incl hash+cmp+emit           5.1 ns/\n[…]\nw identity tests for the\ndoc-context key and encode_piece_into paths), pretokie lib 69/69,\nowt_compare openai-community/gpt2 20342/20342 and tokiers/DeepSeek-V3 vs\ndeepseek-ai/DeepSeek-V3 20342/20342.",
          "is_bot": false,
          "headline": "perf: cache-first encoder loop — 2x faster per-piece encode step",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T07:26:21Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "b0402e7d6f79f36aafecc8bcf2b6739236adf6d9",
          "body": "Replace the DAAC backtracking walk with a direct rank-based BPE merge\nloop (gigatoken-style) for pieces <= 32 bytes that miss both the\nPretokenCache and the token_cache early exit. Longer pieces and the\nencode() entry keep the DAAC walk.\n\nRankPairTable, built in all three constructors from pair_look\n[…]\nnt batch MT   284.6 -> 322.5 MB/s (+13%)\n\nGates: tokie --lib 135 pass, pretokie --lib 69 pass, owt_compare\n20342/20342 for openai-community/gpt2 and for tokiers/DeepSeek-V3 vs\ndeepseek-ai/DeepSeek-V3.",
          "is_bot": false,
          "headline": "perf: rank-table merge core for the short-piece cache-miss path",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T04:56:42Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "b7a694193b668920343a5181459914695a2ccfa5",
          "body": null,
          "is_bot": false,
          "headline": "chore: untrack .claude session directory swept in by the version bump",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T04:41:41Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "17e102f2d99be457ab0f6e576dc0ae2cda935b5e",
          "body": null,
          "is_bot": false,
          "headline": "Bump version to v0.1.2",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T04:41:17Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "af9f12c61cb6c6bf1a853677ff62416965a4209e",
          "body": null,
          "is_bot": false,
          "headline": "build: gate owt_compare example behind the build feature",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T04:40:27Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "fb6e2379423e7190ace0ff370bcd9c16819a27bb",
          "body": "… stale .tkz\n\nDev builds from a git checkout now append a source hash (build.rs, FNV-1a\nover src/**/*.rs) to the compiled cache filename, so semantic changes\nwithout a version bump invalidate the cache automatically. crates.io\nbuilds emit no discriminator - release filenames are unchanged.\n\nAlso: TOKIE_NO_COMPILED_CACHE=1 skips compiled-cache read/write entirely\n(for regen verification scripts), and 'tokie cache clear' removes\ntokenizer.compiled-*.tkz artifacts from the HF hub cache.",
          "is_bot": false,
          "headline": "fix: build-discriminated compiled cache key so dev builds never serve…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T02:50:11Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "a0e3c9f1b9235cddeb804e0239edd246ca54a62c",
          "body": "…metaspace prepend\n\nThree classes of parity gaps held 13 tokiers repos out of the v13 regen:\n\n- Added-token flags were ignored: lstrip/rstrip/normalized/single_word now\n  ride along as AddedTokenSpec, serialized in the v13 ADDED_TOKENS flags\n  byte (bits 2-5; old files read back with pre-flag behavi\n[…]\nack\nrepos plus TinyLlama, xlm-roberta and all-mpnet as regression guards.\nFull regen dry-run passes 94/94; the held-back repos are re-verified\n(1MB enwik8 + probe, exact vs HF) and re-uploaded as v13.",
          "is_bot": false,
          "headline": "fix: HF-exact added-token semantics — flags, id remapping, per-model …",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-24T02:20:51Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "732e272c6c3aa8efb9e0e574af381b2aed371cf5",
          "body": "…ilter\n\nSeven BERT-family repos pad to a fixed length by default, which the\nverification misread as mismatches; with padding disabled all seven\nverify exactly and are uploaded as v13 (81/94 total).",
          "is_bot": false,
          "headline": "scripts: disable HF padding in regen verification, add TOKIERS_ONLY f…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T22:18:33Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "9ecbbd3349392bd61c5cb126d8d3423cfd694c74",
          "body": "Builds each repo's tokenizer.tkz from its own tokenizer.json (added and\nspecial tokens included via the hub load path), verifies the RELOADED\nfile against HuggingFace tokenizers on 1MB enwik8 plus an added-token\nprobe string (which only passes if added tokens survived inside the\n.tkz), and uploads only repos that pass.",
          "is_bot": false,
          "headline": "scripts: tokiers v13 regeneration with per-repo HF verification",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T22:12:26Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "a4cd7fedfdf28231bb0936acc9b32a731c846216",
          "body": null,
          "is_bot": false,
          "headline": "Bump version to v0.1.1",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T22:09:43Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "5a7a9f3cc005893d61b438e3e155b81922e24073",
          "body": "…flow",
          "is_bot": false,
          "headline": "fix: resolve leftover merge conflict markers in speed-regression work…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T22:04:11Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "1400f5b389fe592dd9bc43a26f6e5bd17a5ca88a",
          "body": "All benchmarks re-run on Apple M3 with tokie 0.1.0: single-string encode\nis now 25-154x vs HF tokenizers (was 15-29x), 17-42x vs tiktoken, batch\n17-36x, warm loads 13-44x (2.8-23ms). kitoken and tiktoken bars\nre-measured on the same machine. generate_charts.py data blocks updated\n(the script hardcodes chart data; it does not read\nbenchmark_results.json). Adds gigatoken to the acknowledgements.",
          "is_bot": false,
          "headline": "docs: refresh benchmark numbers and charts for 0.1.0",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T22:03:02Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "1299cdf8f12a22e94ad37a569348fa9d9af38433",
          "body": "The format predated added-token support, so every .tkz load re-fetched\ntokenizer.json just to reconstruct the added-token matcher (and the\ncompiled load cache needed a json sidecar). v13 grows the header to 96\nbytes with a seventh section pair and serializes (id, flags, bytes)\nentries — files are now self-contained, loaders skip the json fetch when\nthe section is present, and the compiled cache drops its sidecar.\nReaders still accept v10-v12; those fall back to the json fetch as\nbefore.",
          "is_bot": false,
          "headline": "feat: .tkz v13 stores added and special tokens",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T21:56:50Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "d65b3d3508ee68d8119ce4fa1b71d550071da8b5",
          "body": "# Conflicts:\n#\t.github/workflows/speed-regression.yml\n#\tCargo.lock\n#\tcrates/tokie-python/Cargo.toml\n#\tcrates/tokie-python/pyproject.toml\n#\tcrates/tokie/Cargo.toml",
          "is_bot": false,
          "headline": "Merge remote-tracking branch 'origin/main'",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T21:50:19Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "063202688e6a4cdfa7830b3b35e51d1667ac8c93",
          "body": "pretokie 0.0.4 -> 0.1.0 (first publish since 0.0.3; tokie now consumes\nthe workspace crate by path+version), tokie + tokie-python -> 0.1.0.\nRelease workflow publishes pretokie before tokie.",
          "is_bot": false,
          "headline": "Bump version to v0.1.0",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T21:45:37Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "b97cd4042d84c6b97a556d6ecd318c5dc83f1041",
          "body": "A cached artifact compiled before a format/build change could be served\nafter it (the v12 reader accepts v11 files), which silently reintroduced\nthe pre-charsmap xlm-roberta normalizer on the merged branch. The cache\nfilename now carries both the crate version and the .tkz format version;\nstale artifacts are simply never looked up. Also sweeps old un-keyed\ncache files from the wild via the name change itself.",
          "is_bot": false,
          "headline": "fix: key the compiled load cache by .tkz format version",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T21:21:27Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "959727ddac7fc1ec34dfeccf3cab02adde127704",
          "body": null,
          "is_bot": false,
          "headline": "Merge branch 'perf/hotpath' into fix/pretokenizer-parity",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T21:17:40Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "3244ac53c810cae4684a491522ed26876b8f6711",
          "body": null,
          "is_bot": false,
          "headline": "chore: lockfile for rust-numpy dependency",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:57:12Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "9b871bd54ea00f818bf83c90d5718d8e6095905d",
          "body": "…ation to 1.3 GB/s\n\nNon-ASCII chars are now classified in-mask per config semantics (letter/\nnumber/punct via Core's exact unicode predicates), joining the same u64\nboundary algebra as ASCII, with a prev-char walk-back claiming straddling\nbytes. Deferral to the scalar path shrinks to the genuinely u\n[…]\nk fuzz strings x 7 configs, owt_sample piece-for-piece x 7,\n20342/20342 vs HF on gpt2 and DeepSeek-V3.\n\ngpt2 on M3, 10MB OWT, single thread: pretokenize-only 470 (Core) ->\n860 (v1 mask) -> ~1300 MB/s.",
          "is_bot": false,
          "headline": "perf: in-mask unicode classification + contraction fixups, pretokeniz…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:55:00Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "ff80c45c4c3be8ffb4d784e6f6310b853754f36d",
          "body": "…d added tokens\n\nHF's Prepend(\"▁\") + Replace(\" \"→\"▁\") normalizer sequence (Llama,\nMistral, TinyLlama) prepends unconditionally, and added-token splitting\nnormalizes each segment independently — so \"Hello </s> world\" emits ▁\ntokens on both sides of the special (▁Hello ▁ </s> ▁ ▁world). tokie\nskipped \n[…]\nnstead of ▁▁spaces). Present in released tokie\n0.0.10 as well.\n\nEmpty input now stays empty (HF Prepend is a no-op there). Adds an\nignored accuracy regression test covering segment shapes around </s>.",
          "is_bot": false,
          "headline": "fix: SP metaspace prepend is unconditional, restoring HF parity aroun…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:29:24Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "d1660dcb50e92a2c41f451d2ac85e779e5608811",
          "body": "Batch calls (encode_batch, count_tokens_batch, encode_batch_flat) now\nsplit work into cpus*4 byte-balanced chunks that one worker per CPU\nclaims from a shared atomic index, instead of one fixed chunk per core —\non asymmetric P/E parts a slow efficiency core no longer sets the wall\ntime for the whole\n[…]\n0MB throughput is unchanged (~390 MB/s count-batch): profiling\nshows the wall is shared-cache/DRAM contention from BPE table probes,\nnot scheduling — that is core-encoder territory, not orchestration.",
          "is_bot": false,
          "headline": "perf: work-stealing batch scheduler + pooled long-lived pretoken caches",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:24:04Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "aaf8fb6aea7e4c9ee0ab5cc3bac97ad1a822593a",
          "body": "Rust: encode_batch_flat(texts, add_special_tokens) -> (Vec<u32>, Vec<u64>)\nover the byte-balanced parallel path with per-thread pretoken caches.\nPython: returns (np.uint32 ids, np.uint64 lengths) via rust-numpy\nIntoPyArray — the Rust buffers are handed to numpy without copying and\nno per-document Python objects are materialized, matching gigatoken's\nbulk output contract. Padding is not applied; truncation and special\ntokens are.",
          "is_bot": false,
          "headline": "feat: encode_batch_flat — contiguous u32 ids + per-doc lengths",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:18:35Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "4d43d139d6fb414e56a95269f0a09b50645b91c7",
          "body": null,
          "is_bot": false,
          "headline": "profile: add single-thread cached-encode row to hotpath profiler",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:15:55Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "4a4ec563c57a6ae5ff8a762881fcb677c3e44b44",
          "body": "64-byte batches are classified with NEON into per-byte class bitmasks\n(simdjson-style movemask), piece-start bits are derived with shifted-mask\nalgebra parameterized by PretokConfig, and iteration pops one bit per\npiece. Core<C> stays the scalar ground truth: non-ASCII chars,\napostrophe contractions\n[…]\nross all 7 configs, full owt_sample\npiece-for-piece diff, and 20342/20342 OWT document parity vs HF on both\ngpt2 and DeepSeek-V3.\n\ngpt2 on M3, 10MB OWT: pretokenize-only 470 -> 860 MB/s single-thread.",
          "is_bot": false,
          "headline": "perf: NEON mask-scanner pretokenizer, 1.8x single-thread pretokenization",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T20:14:53Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "36932b91dd17606c9f02aa7235ef1ac0b72fdf53",
          "body": "…er-piece Viterbi\n\nCloses the ~1% OWT parity gap on Precompiled-normalizer models (was 53/5050\nmismatched docs on xlm-roberta; now 0 across 8 SP models).\n\n- charsmap.rs: parse and apply the tokenizer.json precompiled_charsmap blob\n  (darts double-array trie + replacement string), reproducing HF's gr\n[…]\ne, t5-base,\nbge-m3, snowflake-arctic-embed-l-v2.0, jina-embeddings-v3, both Cohere\nmultilingual models, deepset-mxbai-embed-de-large-v1 — all 5050/5050 with\nidentical .tkz round-trips; owt_ suite 7/7.",
          "is_bot": false,
          "headline": "fix: exact SentencePiece parity — precompiled charsmap, f64 scores, p…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T10:28:59Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "83cb46c6234329045e60a63e14bd9e9cf4bbbb38",
          "body": "…ids-only fast path\n\n- pretokie: ascii_letter_run consumes ASCII-letter runs 8 bytes at a time\n  (carry-safe masked-add SWAR, exhaustively tested against the scalar\n  classifier); wired into scan_letters/scan_ascii_letters. Pretokenize\n  435 -> 470 MB/s, single-thread count 113 -> 137 MB/s.\n- tokeni\n[…]\ngatoken: 0.54us).\n\nVerified: all Rust suites green, 43 python binding tests green, OWT\nparity 9909/9909 (48MB, gpt2/qwen3/smollm2/deepseek/bert) and\nowt_compare 20342/20342 (95MB, gpt2 + DeepSeek-V3).",
          "is_bot": false,
          "headline": "perf: SWAR letter scan, byte-balanced batching, compiled load cache, …",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T10:14:29Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "17bcbd40a508e641ffe3e7787463a6d10d0c5347",
          "body": "- PretokenCache: per-thread open-addressing table (32-byte entries, 16-byte\n  key block compared as one memcmp, up to 3 inline token ids). Pretoken\n  frequency is Zipfian, so most pieces resolve in one probe instead of a\n  hash-map lookup + DAAC backtracking.\n- encode_into: append API threaded throu\n[…]\nially under the GIL.\n\ngpt2/owt on M3: encode_batch (Python) 69 -> 203 MB/s from the lazy binding\nalone; cache impact measured next. Includes the tokenizer.rs normalizer\naccessor fix needed to compile.",
          "is_bot": false,
          "headline": "perf: pretoken cache, append-mode encoding, lazy Python encodings",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:54:08Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "08ef22eb80d38c11355a197c2f47e0d3327f5c95",
          "body": "enwik8 is ASCII-heavy XML-escaped Wikipedia and never exercised the\nunicode that caused real parity bugs (zero-width spaces, smart quotes\nbefore newlines, No-category numerics, uppercase contractions). Add\nper-doc comparison vs HF over a 50MB OWT slice (stanford-cs336/owt-sample,\nsplit on <|endoftex\n[…]\npretokenizer family,\nloaded from the original HF repo so tokenizer.json detection is exercised.\nknown_gap_* tests document the xlm-roberta Precompiled-charsmap gap\n(~1% of docs) and are skipped in CI.",
          "is_bot": false,
          "headline": "ci: per-document OpenWebText accuracy tests",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:38:08Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "2193e05de61504c942c3c6e5ad3485a4e28ecd0d",
          "body": "…n rule\n\nDeepSeek's pattern has no contraction alternation and its punct class is\nthe positive [\\p{P}\\p{S}], not a negated class: Cf/Cc chars (U+200B,\nU+00AD, C1 controls) are not punct — they act as the letter rule's\noptional one-char prefix or stand alone, and apostrophe+letters goes\nthrough the ASCII punct-prefix rule ('Toole stays whole). Non-ASCII\n\\p{P}\\p{S} no longer prefixes letters for DeepSeek (AsciiOnly).\n\nFixes the residual ~0.8% per-doc mismatch vs HF on OpenWebText.",
          "is_bot": false,
          "headline": "fix: DeepSeek pretok parity — [\\p{P}\\p{S}] punct class, no contractio…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:38:08Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "2753a16d160384e4a2bb7069367f19de43bc2ef5",
          "body": "… ranges\n\nThe hand-rolled is_unicode_mark table was missing 1015 mark codepoints —\nincluding the entire Gujarati block (U+0A81..U+0AFF), Arabic Extended,\nCyrillic Extended-A, kana voicing marks — and misclassified 126 non-marks\n(e.g. Hangul Jamo, which are Lo letters). The DeepSeek punct-class rewor\n[…]\nt.\n\nThe table is now generated (Unicode 15.1, 310 ranges) by\nscripts/gen_unicode_marks.py with binary search; also adds\nis_punct_or_symbol/is_format_or_private helpers for [\\p{P}\\p{S}]\nclassification.",
          "is_bot": false,
          "headline": "fix: exact Unicode M-category table, replacing incomplete hand-rolled…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:38:08Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "bef33ac1781ef2342cb86ccd69723db8a7dad610",
          "body": "detect_pretokenizer_type only looked for \\p{N}{ inside the same Split\npattern that contains [\\p{L}\\p{M}]+, but DeepSeek-V3 puts \\p{N}{1,3} in\nits own Split stage before the letter pattern, so detection fell through\nto Qwen35 (DigitMode::Single). Scan all Split stages for chunked digits\nbefore classi\n[…]\ng.\n\nVerified per-doc on a 95MB OpenWebText slice: 20342/20342 docs match HF\nvia both the tokiers .tkz path and direct tokenizer.json loading.\nAdds examples/owt_compare.rs for per-doc web-text diffing.",
          "is_bot": false,
          "headline": "fix: DeepSeek-V3 dispatched to Qwen35 pretok, encoding digits singly",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:25:37Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "0a45f04ba248e86f5a426117e2a9c3d2009709cb",
          "body": "- check_contraction: drop the followed-by-letter guard; the regex\n  '(?i:[sdmt]|ll|ve|re) has no lookahead (O'Toole -> O 'T oole,\n  don'ts -> don 't s)\n- scan_digits: \\p{N} runs include non-ASCII numerics (No/Nl: 1/2/3, 12 1)\n  in Unlimited and Chunked3 modes; GPT-2 space branch now attaches\n  ' ' +\n[…]\ntiktoken\n1500/1500. Note: pretokie must be published before the next tokie\nrelease; DeepSeek-V3 digit-chunking dispatch is broken on this branch\nindependent of this change (382/4068 before and after).",
          "is_bot": false,
          "headline": "fix: four pretokenizer parity bugs vs HF on web-text unicode",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-07-23T06:04:04Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "9b78cc552df73c0fa41451c51e08fe94638fa48c",
          "body": "feat: native PyPI support for Pyodide (PyEmscripten)",
          "is_bot": false,
          "headline": "Merge pull request #10 from chonkie-inc/dev",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:42:05Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "286b8591500e2433be6bcb2c0233f9ed1d09604a",
          "body": null,
          "is_bot": false,
          "headline": "bump version",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:36:28Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "ac6cc6f16e405896e3030916388875acc0fdf027",
          "body": "chore: bump version",
          "is_bot": false,
          "headline": "Merge pull request #9 from chonkie-inc/dev",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:19:42Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "2ac4e8a792c2fef3e45db563f186086f0ab1e94a",
          "body": null,
          "is_bot": false,
          "headline": "bump version",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:18:42Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "21df4626999c2a5c138eadd23c692667fab0582c",
          "body": "chore: update publish workflow",
          "is_bot": false,
          "headline": "Merge pull request #8 from chonkie-inc/dev",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:10:05Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "87eb1ec67860dede03e620907c2234fb79079b90",
          "body": null,
          "is_bot": false,
          "headline": "add Python 3.14",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:06:44Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "2afc1bd87056697e5d2d0a343e48805cca4c40d0",
          "body": null,
          "is_bot": false,
          "headline": "update names",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T17:01:54Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "32a7ab7c7c3ea2e30f9b2d8bb4eb1d8dcdce7689",
          "body": null,
          "is_bot": false,
          "headline": "update pyodide workflow",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T16:58:33Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "4b2b8007fabb8192d61be643e467e9c716dbefb0",
          "body": null,
          "is_bot": false,
          "headline": "update publish workflow",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T16:55:26Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "c31428fc481efb96f3bdf19ab202d3929167ed9c",
          "body": "feat: add pyodide compatibility",
          "is_bot": false,
          "headline": "Merge pull request #7 from chonkie-inc/dev",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T16:11:36Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "7a26a13bf2d0e0cc1e8bba32fbd059cf24c4b075",
          "body": "Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>",
          "is_bot": false,
          "headline": "Potential fix for pull request finding",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-28T15:57:30Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "c38233eae3d300ca55b02b76a0c64d2566e15007",
          "body": null,
          "is_bot": false,
          "headline": "bump version",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-26T18:31:11Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "a3afbe03dfbaedeea1cb5c419088f0ff686a0663",
          "body": null,
          "is_bot": false,
          "headline": "update workflow",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-26T18:22:22Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "3958860e97807db511b0e464a5734cb5d4afec54",
          "body": "Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>",
          "is_bot": false,
          "headline": "Potential fix for pull request finding",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-26T18:16:00Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "8ae276cbb26fe4a381a92f3bb329d877a11b9faa",
          "body": "…ntain permissions'\n\nCo-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>",
          "is_bot": false,
          "headline": "Potential fix for pull request finding 'CodeQL / Workflow does not co…",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-26T18:11:20Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "ad25cab28d9089f27fbf331911b48ea86bdbe259",
          "body": null,
          "is_bot": false,
          "headline": "add Emscripten wheel",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-26T18:07:19Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "968f9a21ee360e0704a0be0c1468c90c7e1b6f66",
          "body": "support python 3.10",
          "is_bot": false,
          "headline": "Merge pull request #5 from chonkie-inc/dev",
          "author_name": "Hafedh Hichri",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-05T18:14:48Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "2da03f206cca318f96898cea3f4294ce3f14477a",
          "body": null,
          "is_bot": false,
          "headline": "fix workflow",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-05T18:09:33Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "823df386c0d33fc05e9c2381ee12e2b9d44a16b7",
          "body": null,
          "is_bot": false,
          "headline": "bump version",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-05T18:05:28Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "99a352a5af8073ffd570670ad256038f0029a528",
          "body": null,
          "is_bot": false,
          "headline": "support python 3.10",
          "author_name": "chonk-lain",
          "author_login": "chonk-lain",
          "committed_at": "2026-05-05T17:52:22Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "f783c82eeef1858fd09937f8a70f3feebdd61d28",
          "body": "Chunking enwik8 into 1KB pieces created false positives at chunk\nboundaries because BPE tokenization is context-dependent. Now encodes\nthe first 1MB as a single string, matching how the accuracy test suite\nworks. GPT-2 is 100% accurate (286273/286273 tokens match).",
          "is_bot": false,
          "headline": "fix: verify full text instead of chunks to avoid false BPE mismatches",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T03:20:53Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "f67ba6d0798afe1235d82072991b5f8f7638a3dd",
          "body": null,
          "is_bot": false,
          "headline": "fix: speed regression CI fails on Cargo.lock conflict during checkout",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T02:35:33Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "987a8e80aa613800a59036b32d89457e08b418ce",
          "body": "The accuracy test suite was only running on push to main. Now it also\nruns on pull_request, so accuracy regressions are caught before merge.\nAlso update the feature gate from hf to build since accuracy.rs imports\ntokenizers and tiktoken-rs which are now behind the build feature.",
          "is_bot": false,
          "headline": "ci: run tokenizer accuracy tests on PRs, update feature gate to build",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T02:23:28Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "f72b0c8f2512e0143de02cea9f9bf04e8837baca",
          "body": "Download and cache enwik8 (~100MB), split into ~1KB chunks, and compare\ntoken IDs against the reference backend. This gives ~80K test cases\ninstead of 8, catching edge cases around unicode and special characters.\n\nAlso improve CLI output: show first divergence point per mismatch,\nlimit to 5 shown with a count of remaining.",
          "is_bot": false,
          "headline": "feat: use enwik8 for verification instead of hardcoded sentences",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:55:09Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "1cf2e8214d6a4d96c623499f0c78496a5678ae64",
          "body": null,
          "is_bot": false,
          "headline": "chore: update Cargo.lock for tokie-cli",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:43:54Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "0ac3e6421450e1c6043466a69353c36a78a2a1f7",
          "body": null,
          "is_bot": false,
          "headline": "feat: add tokie-cli crate with convert and verify commands",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:38:43Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "378539a3c25f88707d65dbf6a9d8aec071196f09",
          "body": null,
          "is_bot": false,
          "headline": "feat: add upload function for tokiers/ HF org",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:37:50Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "207884cb56fae28eb4b9656a66cf7f2515708b87",
          "body": null,
          "is_bot": false,
          "headline": "feat: add verify function with auto-detected reference backend",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:36:50Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "478a8e26f70fe4eaa6eccc29e380d08a75d21961",
          "body": null,
          "is_bot": false,
          "headline": "feat: add build module with convert function",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:36:12Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "1018920192a09b2f060950fd36965ddde2f00ea2",
          "body": "… deps",
          "is_bot": false,
          "headline": "feat: add build feature flag, move tokenizers/tiktoken-rs to optional…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:35:35Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "40edb989e332572ca378365ab2624a8f97aa8e7f",
          "body": null,
          "is_bot": false,
          "headline": "docs: add implementation plan for build module and tokie-cli",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:29:46Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "4c77e5a5d4c3e71403a021a6d3d5c906a7a64a1e",
          "body": "Spec for adding a `build` feature to the core crate with convert/verify/upload\nfunctions, plus a tokie-cli crate that wires them to a `tokie` CLI binary.",
          "is_bot": false,
          "headline": "docs: add design spec for build module and tokie-cli",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-04-22T00:24:32Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "06106e6c3b275b1e98a8e3742f1e411092bc2f9b",
          "body": "* Refactor pretokie: trait-based Core replaces 7 duplicated iterators\n\nExtract shared logic from GPT-2, CL100K, O200K, Voyage, SmolLM,\nDeepSeek, and Qwen pretokenizers into a generic Core<C: PretokConfig>\nstruct. Each pretokenizer is now a zero-sized config type + type alias.\n\nThe compiler monomorph\n[…]\nases preserve Gpt2::new(), etc.)\n- BERT stays standalone (different paradigm)\n\n* Update pretokie README benchmarks, bump to v0.0.4\n\nRefresh benchmark numbers after trait-based refactor (307-351 MB/s).",
          "is_bot": false,
          "headline": "Refactor pretokie: trait-based Core replaces 7 duplicated iterators (#3)",
          "author_name": "Bhavnick @ chonkie.ai",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T23:41:11Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "867a8139a0fac807e8d00b32adec0df36d57d3af",
          "body": "- Flexoki paper bg (#FFFCF0), green tokie (#66800B), amber kitoken (#AD8301)\n- Add kitoken bars to hero, BPE, and WordPiece charts\n- Omit kitoken from SentencePiece (inaccurate output)\n- Fix loading chart: use cl100k consistently (was mixing BERT + cl100k)\n- tiktoken chart: Flexoki dark (#1C1B1A) with vertical bars\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Regenerate charts with proper Flexoki theme and kitoken bars",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T08:35:56Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "033ff235dcd0c996cd24989b5672197f69777401",
          "body": "Restore cream background, olive-green tokie bars, gray competitor bars,\n\"Nx slower\" annotations, and dark-theme tiktoken vertical bar chart.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Fix chart styling to match original design",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T08:25:17Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "b49cb97adb3496eb81fd7dd4072e7e0889ea19b0",
          "body": "Re-run all benchmarks on Apple M3 Pro with latest code. Key changes:\n- tokie vs HF: 15-29x faster (up from 10-20x claimed)\n- tokie vs tiktoken: 3.5-8.3x faster (up from 3.5-5.6x)\n- pretokie throughput: 304-340 MB/s (down from 370-407 MB/s due to\n  accuracy fixes in Unicode/punctuation handling)\n- Re\n[…]\nh_for_readme.py and generate_charts.py scripts\n- Add pretokie verify_vs_hf.rs example\n- Update version references to v0.0.8/v0.0.3\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Update benchmarks, charts, and READMEs with fresh numbers",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T08:12:44Z",
          "body_truncated": true,
          "is_coding_agent": true
        },
        {
          "oid": "6431cf3348cf34c27bd4d012445c4df2ceb8ea1a",
          "body": "Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Switch pretokie dependency to crates.io v0.0.3",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T07:37:27Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "473bb2b3734908f484566c1e1dfc70cac2a545e4",
          "body": "Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Bump pretokie to v0.0.3, tokie to v0.0.8",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T07:34:59Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "c1027e057a997e834a6749e251fcb1ffbbaeeabd",
          "body": "HuggingFace WordPiece defaults to max_input_chars_per_word=100, mapping\nany word longer than 100 characters to [UNK]. tokie was missing this\ncheck, causing mismatches on long single-word inputs.\n\nParse the limit from tokenizer.json, enforce it in encode(), and\nserialize it in .tkz format (backward compatible — old files default\nto 100).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Add WordPiece max_input_chars_per_word limit",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T07:34:21Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "8e7a98f7de49f78a3f8edd9a6c42c8552097c9d3",
          "body": "Improve whitespace, punctuation, and Unicode handling in CL100K, O200K,\nGPT-2, SmolLM, DeepSeek, Qwen, and Voyage pretokenizers. All iterators\nnow match HuggingFace tokenizer output on 74 models tested against\nenwik8.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>",
          "is_bot": false,
          "headline": "Fix pretokenizer accuracy across all 7 specialized iterators",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-30T07:34:05Z",
          "body_truncated": false,
          "is_coding_agent": true
        },
        {
          "oid": "1ad0525e762882f18c991c70aeb99d2c52463e0f",
          "body": "When spaces are followed by newlines, merge all whitespace into one\npiece (implements \\s*[\\r\\n]+ correctly). Also handles \"  \\n\\n\" pattern.\n\nProgress: Llama 3.2, BERT, Gemma 3 now 100% correct on enwik8 100KB.\nQwen3/Voyage fail at token 16200 (Thai mark handling).\nDeepSeek fails at token 32 (separate issue).",
          "is_bot": false,
          "headline": "Fix CL100K whitespace: merge spaces+newlines into single piece",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T08:19:11Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "10db3b0123a6eaeac8353e2a745df916887a7ff7",
          "body": "Three fixes across all CL100K-style pretokenizers (Cl100k, Voyage,\nDeepSeek, Qwen, O200k):\n\n1. Add is_punct_char (excludes ALL whitespace) vs is_prefix_char\n   (excludes only newlines). Punct scanning now correctly stops at\n   spaces/tabs. Fixes: =\"http splitting as [\"=\", \"\\\"http\"] instead\n   of [\"=\n[…]\nnct check uses is_punct_char instead of is_prefix_char\n   to avoid treating space+space as space+punct.\n\nProgress: Llama 3.2/Qwen3/Voyage now match HF until token ~1900\n(previously failed at token 5).",
          "is_bot": false,
          "headline": "Fix CL100K-style pretokenizer punct grouping and space handling",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T08:14:06Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "f95aea0b7968d25752f534898587fed49c8883ca",
          "body": "GPT-2 regex ` ?[^\\s\\p{L}\\p{N}]+` means space merges with following\npunctuation (e.g. ' <' is one piece). pretokie was emitting space\nstandalone before punctuation. Fixed for Gpt2 and SmolLM.\n\nAlso updated pretokie README with benchmarks and tokie branding.\n\nCI accuracy failures still need investigation — some may be pre-existing\nencoder issues unrelated to the pretokenizer refactor.",
          "is_bot": false,
          "headline": "Fix GPT-2/SmolLM pretokenizer: space prefixes punctuation",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:57:00Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "7a4627d9fbd725eaa83889055d3b8197648dbd6e",
          "body": null,
          "is_bot": false,
          "headline": "Update pretokie README with benchmarks, badges, and tokie branding",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:52:06Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "36d5c5b8e3c569036ba6c161df0c0f56927a27ad",
          "body": "- Replace catch-all is_unicode_punct fallback with proper Unicode P* category\n  check using unicode-general-category crate\n- Symbols like ° (So), © (So), ± (Sm) no longer incorrectly classified as punct\n- Fixes BERT/WordPiece accuracy on text containing degree symbols, copyright, etc.\n- pretokie v0.0.2 published to crates.io\n- tokie updated to depend on pretokie 0.0.2",
          "is_bot": false,
          "headline": "Fix BERT Unicode punctuation classification, publish pretokie v0.0.2",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:47:00Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "6bd1fb729e61f15aaf8202a0d63ea98a54a4c633",
          "body": null,
          "is_bot": false,
          "headline": "Remove kitoken comparison from tagline",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:30:43Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "dd2be967f7f65b3983f6dd55348013d6445c0253",
          "body": null,
          "is_bot": false,
          "headline": "Link kitoken SentencePiece issue in benchmark footnote",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:28:02Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "065f07f6cfd6bda50e1e21b71d71df29e1b94425",
          "body": "…ma 3\n\nVerified against HuggingFace: tokie matches exactly (3,156 tokens on 10KB),\nkitoken diverges at token 93 producing 13% more tokens (3,575). The apparent\n1.7x speed advantage was from skipping merges, not algorithmic superiority.\n\nUpdated README and charts to reflect this. SentencePiece chart now shows\ntokie vs HF only (1.3x faster, 100% correct).",
          "is_bot": false,
          "headline": "Fix SentencePiece benchmark: kitoken produces incorrect output on Gem…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:26:39Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "443769a4e4357730b7525daf294025e618bd52b4",
          "body": "- Horizontal bars, flexoki paper background (#FFFCF0), green (#66800B) for tokie\n- Amber (#AD8301) for kitoken, gray for other competitors\n- Separate charts: hero (all competitors), BPE, WordPiece, SentencePiece, loading\n- SentencePiece chart honestly shows kitoken winning (1.7x faster)",
          "is_bot": false,
          "headline": "Regenerate benchmark charts with flexoki theme matching chunk style",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:19:16Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "d24d544dc01bfa2adf253e8227bce3ee0cd35134",
          "body": "…ackground)",
          "is_bot": false,
          "headline": "Fix benchmark chart style to match original (horizontal bars, cream b…",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:15:09Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "69dee7d3ddfb30766b76bebbc8d082b810372315",
          "body": "- Regenerate all benchmark charts including kitoken and tiktoken numbers\n- Add new tiktoken comparison chart (cl100k, o200k)\n- Update descriptive text with honest numbers: 10-20x vs HF, 1.4-3.6x vs kitoken\n- Acknowledge SentencePiece weakness (kitoken 1.7x faster on Gemma 3)",
          "is_bot": false,
          "headline": "Update benchmark charts and README with kitoken/tiktoken comparisons",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:10:30Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "72cff0a8fbb6e33f6caa9d1dc31bfbc57d21470f",
          "body": "- Add pretokie crate: 8 hand-coded zero-allocation pretokenizer iterators\n  (Gpt2, Cl100k, O200k, Bert, Voyage, SmolLM, DeepSeek, Qwen) at 340-380 MB/s\n- Add regex fallback behind feature flag (pretokie v0.0.1 published to crates.io)\n- Replace tokie's config-driven Pretok/PretokIter with Pretokenize\n[…]\nure-gated)\n- Wire regex fallback into hf.rs for unknown tokenizer.json patterns\n- Update README benchmarks: tokie vs HuggingFace vs kitoken vs tiktoken\n- Rename Gpt2Digits to SmolLM across both crates",
          "is_bot": false,
          "headline": "Add pretokie crate with specialized pretokenizers, replace old lexer",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-29T07:07:58Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "af25a17fc4a5f3c606cd03bd791979eea7a385de",
          "body": "In models like Gemma, both a byte-fallback token (e.g., <0x3C> = id 277)\nand a real character token (e.g., '<' = id 235322) map to the same byte.\nThe encoder must use the real token ID to match merge rules.\n\nFix in three places:\n- hf.rs: track byte_fallback_ids during JSON loading, pass to encoder\n-\n[…]\nut construction\n- serde.rs: detect byte-fallback range during .tkz deserialization using\n  merge-operand analysis and contiguous-range heuristic\n\n74/74 accuracy tests pass including both Gemma models.",
          "is_bot": false,
          "headline": "Fix SentencePiece byte-fallback token collisions (Gemma accuracy)",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-26T21:14:53Z",
          "body_truncated": true,
          "is_coding_agent": false
        },
        {
          "oid": "0c8b039a832626265f325f3168a4412091527ea3",
          "body": "- New PretokType::Qwen35 for Qwen3.5's pattern (combining marks + single digits)\n- Distinguish Qwen3.5 from DeepSeek via \\p{N}{ digit-group detection\n- Qwen3 uses existing CL100K pretokenizer (same pattern)\n- Add 5 accuracy tests (Qwen3 0.6B/8B/Coder-30B, Qwen3.5 0.8B/4B)\n- Add all 13 Qwen3/3.5 sizes to regenerate, upload, and populate scripts\n- Update README: 70 → 75 verified models\n- Fix check_models to use google/ for Gemma (needs sentencepiece model file)",
          "is_bot": false,
          "headline": "Add Qwen3/3.5 tokenizer support with new Qwen35 pretokenizer type",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-26T20:37:28Z",
          "body_truncated": false,
          "is_coding_agent": false
        },
        {
          "oid": "200e45836cf46d3f9e97444248c5492787bf8318",
          "body": "Meta Llama and Google Gemma repos require auth tokens which CI\ndoesn't have. Use tokiers/ mirrors (which contain the original\ntokenizer.json) for the HF reference side too.",
          "is_bot": false,
          "headline": "Fix CI: use tokiers/ repos for gated models (Llama, Gemma)",
          "author_name": "Bhavnick @ Chonkie",
          "author_login": "chonknick",
          "committed_at": "2026-03-26T07:11:22Z",
          "body_truncated": false,
          "is_coding_agent": false
        }
      ],
      "releases_count": 14,
      "commits_last_year": 168,
      "latest_release_at": "2026-07-24T09:38:18Z",
      "latest_release_tag": "v0.1.3",
      "releases_from_tags": false,
      "days_since_last_push": 0,
      "active_weeks_last_year": 10,
      "days_since_latest_release": 0,
      "mean_days_between_releases": 13.4
    },
    "community": {
      "has_readme": true,
      "has_license": true,
      "has_description": true,
      "has_contributing": false,
      "health_percentage": 37,
      "has_issue_template": false,
      "has_code_of_conduct": false,
      "has_pull_request_template": false
    },
    "ecosystem": {
      "packages": [
        {
          "name": "tokie",
          "exists": true,
          "license": "MIT OR Apache-2.0",
          "keywords": [
            "tokenizer",
            "nlp",
            "wordpiece",
            "bpe",
            "transformers",
            "science",
            "text-processing"
          ],
          "ecosystem": "crates",
          "matches_repo": false,
          "registry_url": "https://crates.io/crates/tokie",
          "is_deprecated": false,
          "latest_version": "0.1.3",
          "repository_url": "https://github.com/chonkie-inc/tokie",
          "versions_count": 16,
          "total_downloads": 771,
          "dependents_count": null,
          "deprecation_note": null,
          "maintainers_count": null,
          "monthly_downloads": 159,
          "first_published_at": "2026-03-23T03:51:41.367913Z",
          "latest_published_at": "2026-07-24T09:58:00.150709Z",
          "latest_version_yanked": false,
          "days_since_latest_publish": 0
        },
        {
          "name": "pretokie",
          "exists": true,
          "license": "MIT OR Apache-2.0",
          "keywords": [
            "tokenizer",
            "nlp",
            "bpe",
            "transformers",
            "pretokenizer",
            "text-processing"
          ],
          "ecosystem": "crates",
          "matches_repo": false,
          "registry_url": "https://crates.io/crates/pretokie",
          "is_deprecated": false,
          "latest_version": "0.1.3",
          "repository_url": "https://github.com/chonkie-inc/tokie",
          "versions_count": 6,
          "total_downloads": 9489,
          "dependents_count": null,
          "deprecation_note": null,
          "maintainers_count": null,
          "monthly_downloads": 3103,
          "first_published_at": "2026-03-29T05:55:21.853696Z",
          "latest_published_at": "2026-07-24T09:57:39.903575Z",
          "latest_version_yanked": false,
          "days_since_latest_publish": 0
        },
        {
          "name": "tokie",
          "exists": true,
          "license": "MIT OR Apache-2.0",
          "keywords": [
            "tokenizer",
            "tokenization",
            "bpe",
            "wordpiece",
            "sentencepiece",
            "bert",
            "gpt2",
            "llama",
            "nlp",
            "embedding",
            "rag",
            "transformers",
            "huggingface",
            "fast",
            "rust",
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: Apache Software License",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.10",
            "Programming Language :: Python :: 3.11",
            "Programming Language :: Python :: 3.12",
            "Programming Language :: Python :: 3.13",
            "Programming Language :: Python :: 3.14",
            "Programming Language :: Python :: Implementation :: CPython",
            "Programming Language :: Rust",
            "Topic :: Scientific/Engineering :: Artificial Intelligence",
            "Topic :: Text Processing",
            "Topic :: Text Processing :: Linguistic",
            "Typing :: Typed"
          ],
          "ecosystem": "pypi",
          "matches_repo": false,
          "registry_url": "https://pypi.org/project/tokie/",
          "is_deprecated": false,
          "latest_version": "0.1.3",
          "repository_url": "https://github.com/chonkie-inc/tokie",
          "versions_count": 14,
          "total_downloads": null,
          "dependents_count": null,
          "deprecation_note": null,
          "maintainers_count": null,
          "monthly_downloads": null,
          "first_published_at": "2026-03-23T06:46:56.771182Z",
          "latest_published_at": "2026-07-24T09:42:49.945457Z",
          "latest_version_yanked": null,
          "days_since_latest_publish": 0
        }
      ]
    },
    "popularity": {
      "forks": 5,
      "stars": 49,
      "watchers": 0,
      "fork_history": {
        "days": [
          {
            "date": "2026-04-02",
            "count": 1
          },
          {
            "date": "2026-04-23",
            "count": 1
          },
          {
            "date": "2026-05-22",
            "count": 1
          },
          {
            "date": "2026-06-17",
            "count": 1
          },
          {
            "date": "2026-07-08",
            "count": 1
          }
        ],
        "complete": true,
        "collected": 5,
        "total_forks": 5
      },
      "star_history": null,
      "open_issues_and_prs": 3
    },
    "ai_readiness": {
      "has_nix": false,
      "example_dirs": [
        "examples"
      ],
      "has_llms_txt": false,
      "has_dockerfile": false,
      "has_mcp_signal": false,
      "bootstrap_files": [],
      "api_schema_files": [],
      "has_devcontainer": false,
      "typecheck_configs": [],
      "toolchain_manifests": [
        "Cargo.toml",
        "crates/pretokie/Cargo.toml",
        "crates/tokie-cli/Cargo.toml",
        "crates/tokie-python/Cargo.toml",
        "crates/tokie/Cargo.toml"
      ],
      "largest_source_bytes": 90007,
      "source_files_sampled": 96,
      "oversized_source_files": 4,
      "agent_instruction_files": [],
      "agent_instruction_max_bytes": null
    },
    "dependencies": {
      "manifests": [
        "Cargo.toml"
      ],
      "advisories": {
        "error": null,
        "scope": "repository_graph",
        "source": "osv",
        "findings": [
          {
            "name": "number_prefix",
            "direct": false,
            "version": "0.4.0",
            "severity": "unknown",
            "ecosystem": "crates",
            "cvss_score": null,
            "advisory_ids": [
              "RUSTSEC-2025-0119"
            ],
            "fixed_version": null,
            "advisory_count": 1,
            "oldest_advisory_days": 249
          },
          {
            "name": "paste",
            "direct": false,
            "version": "1.0.15",
            "severity": "unknown",
            "ecosystem": "crates",
            "cvss_score": null,
            "advisory_ids": [
              "RUSTSEC-2024-0436"
            ],
            "fixed_version": null,
            "advisory_count": 1,
            "oldest_advisory_days": 655
          }
        ],
        "collected": true,
        "malicious": [],
        "truncated": false,
        "by_severity": {
          "unknown": 2
        },
        "advisory_count": 2,
        "affected_count": 2,
        "assessed_count": 281,
        "malicious_count": 0,
        "assessed_package": null,
        "unassessed_count": 26,
        "direct_affected_count": 0
      },
      "ecosystems": [
        "crates"
      ],
      "dependencies": [
        {
          "name": "regex-automata",
          "manifest": "crates/pretokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.4"
        },
        {
          "name": "unicode-general-category",
          "manifest": "crates/pretokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.0"
        },
        {
          "name": "tokie",
          "manifest": "crates/tokie-cli/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": null
        },
        {
          "name": "clap",
          "manifest": "crates/tokie-cli/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "4"
        },
        {
          "name": "hf-hub",
          "manifest": "crates/tokie-cli/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.4.3"
        },
        {
          "name": "tokie-core",
          "manifest": "crates/tokie-python/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": null
        },
        {
          "name": "pyo3",
          "manifest": "crates/tokie-python/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.29"
        },
        {
          "name": "numpy",
          "manifest": "crates/tokie-python/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.29"
        },
        {
          "name": "crc32fast",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.4"
        },
        {
          "name": "daggrs",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.1.1"
        },
        {
          "name": "foldhash",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.1"
        },
        {
          "name": "smallvec",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.13"
        },
        {
          "name": "hf-hub",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.4.3"
        },
        {
          "name": "memchr",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "2.7"
        },
        {
          "name": "chunk",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.9"
        },
        {
          "name": "pretokie",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.1.3"
        },
        {
          "name": "icu_normalizer",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.5"
        },
        {
          "name": "serde",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.0"
        },
        {
          "name": "serde_json",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.0"
        },
        {
          "name": "unicode-general-category",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.0"
        },
        {
          "name": "unicode-segmentation",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "1.12"
        },
        {
          "name": "base64",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.22"
        },
        {
          "name": "tokenizers",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.22"
        },
        {
          "name": "tiktoken-rs",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "0.6"
        },
        {
          "name": "ureq",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "2"
        },
        {
          "name": "zip",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "2"
        },
        {
          "name": "dirs-next",
          "manifest": "crates/tokie/Cargo.toml",
          "ecosystem": "crates",
          "version_constraint": "2"
        }
      ],
      "all_dependencies": {
        "error": null,
        "source": "github-sbom",
        "packages": [
          {
            "name": "base64",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "base64",
            "direct": true,
            "version": "0.13.1",
            "ecosystem": "crates"
          },
          {
            "name": "base64",
            "direct": true,
            "version": "0.21.7",
            "ecosystem": "crates"
          },
          {
            "name": "base64",
            "direct": true,
            "version": "0.22.1",
            "ecosystem": "crates"
          },
          {
            "name": "chunk",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "chunk",
            "direct": true,
            "version": "0.9.2",
            "ecosystem": "crates"
          },
          {
            "name": "clap",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "clap",
            "direct": true,
            "version": "4.6.4",
            "ecosystem": "crates"
          },
          {
            "name": "crc32fast",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "crc32fast",
            "direct": true,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "daggrs",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "daggrs",
            "direct": true,
            "version": "0.1.1",
            "ecosystem": "crates"
          },
          {
            "name": "dirs-next",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "dirs-next",
            "direct": true,
            "version": "2.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "foldhash",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "foldhash",
            "direct": true,
            "version": "0.1.5",
            "ecosystem": "crates"
          },
          {
            "name": "hf-hub",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "hf-hub",
            "direct": true,
            "version": "0.4.3",
            "ecosystem": "crates"
          },
          {
            "name": "icu_normalizer",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "icu_normalizer",
            "direct": true,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_normalizer",
            "direct": true,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "memchr",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "memchr",
            "direct": true,
            "version": "2.8.3",
            "ecosystem": "crates"
          },
          {
            "name": "numpy",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "numpy",
            "direct": true,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "pretokie",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "pyo3",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "pyo3",
            "direct": true,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "regex-automata",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "regex-automata",
            "direct": true,
            "version": "0.4.16",
            "ecosystem": "crates"
          },
          {
            "name": "serde",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "serde",
            "direct": true,
            "version": "1.0.229",
            "ecosystem": "crates"
          },
          {
            "name": "serde_json",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "serde_json",
            "direct": true,
            "version": "1.0.151",
            "ecosystem": "crates"
          },
          {
            "name": "smallvec",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "smallvec",
            "direct": true,
            "version": "1.15.2",
            "ecosystem": "crates"
          },
          {
            "name": "tiktoken-rs",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "tiktoken-rs",
            "direct": true,
            "version": "0.6.0",
            "ecosystem": "crates"
          },
          {
            "name": "tokenizers",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "tokenizers",
            "direct": true,
            "version": "0.22.2",
            "ecosystem": "crates"
          },
          {
            "name": "unicode-general-category",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "unicode-general-category",
            "direct": true,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "unicode-segmentation",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "unicode-segmentation",
            "direct": true,
            "version": "1.13.3",
            "ecosystem": "crates"
          },
          {
            "name": "ureq",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "ureq",
            "direct": true,
            "version": "2.12.1",
            "ecosystem": "crates"
          },
          {
            "name": "zip",
            "direct": true,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "zip",
            "direct": true,
            "version": "2.4.2",
            "ecosystem": "crates"
          },
          {
            "name": "adler2",
            "direct": false,
            "version": "2.0.1",
            "ecosystem": "crates"
          },
          {
            "name": "ahash",
            "direct": false,
            "version": "0.8.12",
            "ecosystem": "crates"
          },
          {
            "name": "aho-corasick",
            "direct": false,
            "version": "1.1.4",
            "ecosystem": "crates"
          },
          {
            "name": "anes",
            "direct": false,
            "version": "0.1.6",
            "ecosystem": "crates"
          },
          {
            "name": "anstream",
            "direct": false,
            "version": "1.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "anstyle",
            "direct": false,
            "version": "1.0.14",
            "ecosystem": "crates"
          },
          {
            "name": "anstyle-parse",
            "direct": false,
            "version": "1.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "anstyle-query",
            "direct": false,
            "version": "1.1.5",
            "ecosystem": "crates"
          },
          {
            "name": "anstyle-wincon",
            "direct": false,
            "version": "3.0.11",
            "ecosystem": "crates"
          },
          {
            "name": "anyhow",
            "direct": false,
            "version": "1.0.104",
            "ecosystem": "crates"
          },
          {
            "name": "arbitrary",
            "direct": false,
            "version": "1.4.2",
            "ecosystem": "crates"
          },
          {
            "name": "autocfg",
            "direct": false,
            "version": "1.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "bit-set",
            "direct": false,
            "version": "0.5.3",
            "ecosystem": "crates"
          },
          {
            "name": "bit-vec",
            "direct": false,
            "version": "0.6.3",
            "ecosystem": "crates"
          },
          {
            "name": "bitflags",
            "direct": false,
            "version": "2.13.1",
            "ecosystem": "crates"
          },
          {
            "name": "bstr",
            "direct": false,
            "version": "1.13.0",
            "ecosystem": "crates"
          },
          {
            "name": "bumpalo",
            "direct": false,
            "version": "3.20.3",
            "ecosystem": "crates"
          },
          {
            "name": "byteorder",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "bytes",
            "direct": false,
            "version": "1.12.1",
            "ecosystem": "crates"
          },
          {
            "name": "cast",
            "direct": false,
            "version": "0.3.0",
            "ecosystem": "crates"
          },
          {
            "name": "castaway",
            "direct": false,
            "version": "0.2.4",
            "ecosystem": "crates"
          },
          {
            "name": "cc",
            "direct": false,
            "version": "1.3.0",
            "ecosystem": "crates"
          },
          {
            "name": "cfg-if",
            "direct": false,
            "version": "1.0.4",
            "ecosystem": "crates"
          },
          {
            "name": "ciborium",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "ciborium-io",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "ciborium-ll",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "clap_builder",
            "direct": false,
            "version": "4.6.2",
            "ecosystem": "crates"
          },
          {
            "name": "clap_derive",
            "direct": false,
            "version": "4.6.4",
            "ecosystem": "crates"
          },
          {
            "name": "clap_lex",
            "direct": false,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "colorchoice",
            "direct": false,
            "version": "1.0.5",
            "ecosystem": "crates"
          },
          {
            "name": "compact_str",
            "direct": false,
            "version": "0.9.1",
            "ecosystem": "crates"
          },
          {
            "name": "console",
            "direct": false,
            "version": "0.15.11",
            "ecosystem": "crates"
          },
          {
            "name": "console",
            "direct": false,
            "version": "0.16.4",
            "ecosystem": "crates"
          },
          {
            "name": "criterion",
            "direct": false,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "criterion",
            "direct": false,
            "version": "0.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "criterion-plot",
            "direct": false,
            "version": "0.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "crossbeam-deque",
            "direct": false,
            "version": "0.8.7",
            "ecosystem": "crates"
          },
          {
            "name": "crossbeam-epoch",
            "direct": false,
            "version": "0.9.20",
            "ecosystem": "crates"
          },
          {
            "name": "crossbeam-utils",
            "direct": false,
            "version": "0.8.22",
            "ecosystem": "crates"
          },
          {
            "name": "crunchy",
            "direct": false,
            "version": "0.2.4",
            "ecosystem": "crates"
          },
          {
            "name": "darling",
            "direct": false,
            "version": "0.20.11",
            "ecosystem": "crates"
          },
          {
            "name": "darling_core",
            "direct": false,
            "version": "0.20.11",
            "ecosystem": "crates"
          },
          {
            "name": "darling_macro",
            "direct": false,
            "version": "0.20.11",
            "ecosystem": "crates"
          },
          {
            "name": "dary_heap",
            "direct": false,
            "version": "0.3.9",
            "ecosystem": "crates"
          },
          {
            "name": "derive_arbitrary",
            "direct": false,
            "version": "1.4.2",
            "ecosystem": "crates"
          },
          {
            "name": "derive_builder",
            "direct": false,
            "version": "0.20.2",
            "ecosystem": "crates"
          },
          {
            "name": "derive_builder_core",
            "direct": false,
            "version": "0.20.2",
            "ecosystem": "crates"
          },
          {
            "name": "derive_builder_macro",
            "direct": false,
            "version": "0.20.2",
            "ecosystem": "crates"
          },
          {
            "name": "dirs",
            "direct": false,
            "version": "6.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "dirs-sys",
            "direct": false,
            "version": "0.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "dirs-sys-next",
            "direct": false,
            "version": "0.1.2",
            "ecosystem": "crates"
          },
          {
            "name": "displaydoc",
            "direct": false,
            "version": "0.2.6",
            "ecosystem": "crates"
          },
          {
            "name": "either",
            "direct": false,
            "version": "1.16.0",
            "ecosystem": "crates"
          },
          {
            "name": "encode_unicode",
            "direct": false,
            "version": "1.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "equivalent",
            "direct": false,
            "version": "1.0.2",
            "ecosystem": "crates"
          },
          {
            "name": "esaxx-rs",
            "direct": false,
            "version": "0.1.10",
            "ecosystem": "crates"
          },
          {
            "name": "fancy-regex",
            "direct": false,
            "version": "0.13.0",
            "ecosystem": "crates"
          },
          {
            "name": "find-msvc-tools",
            "direct": false,
            "version": "0.1.9",
            "ecosystem": "crates"
          },
          {
            "name": "flate2",
            "direct": false,
            "version": "1.1.9",
            "ecosystem": "crates"
          },
          {
            "name": "fnv",
            "direct": false,
            "version": "1.0.7",
            "ecosystem": "crates"
          },
          {
            "name": "form_urlencoded",
            "direct": false,
            "version": "1.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "futures-core",
            "direct": false,
            "version": "0.3.33",
            "ecosystem": "crates"
          },
          {
            "name": "futures-task",
            "direct": false,
            "version": "0.3.33",
            "ecosystem": "crates"
          },
          {
            "name": "futures-util",
            "direct": false,
            "version": "0.3.33",
            "ecosystem": "crates"
          },
          {
            "name": "getrandom",
            "direct": false,
            "version": "0.2.17",
            "ecosystem": "crates"
          },
          {
            "name": "getrandom",
            "direct": false,
            "version": "0.3.4",
            "ecosystem": "crates"
          },
          {
            "name": "half",
            "direct": false,
            "version": "2.7.1",
            "ecosystem": "crates"
          },
          {
            "name": "hashbrown",
            "direct": false,
            "version": "0.17.1",
            "ecosystem": "crates"
          },
          {
            "name": "heck",
            "direct": false,
            "version": "0.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "hermit-abi",
            "direct": false,
            "version": "0.5.2",
            "ecosystem": "crates"
          },
          {
            "name": "http",
            "direct": false,
            "version": "1.4.2",
            "ecosystem": "crates"
          },
          {
            "name": "icu_collections",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_collections",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_locale_core",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_locid",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_locid_transform",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_locid_transform_data",
            "direct": false,
            "version": "1.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "icu_normalizer_data",
            "direct": false,
            "version": "1.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "icu_normalizer_data",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_properties",
            "direct": false,
            "version": "1.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "icu_properties",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_properties_data",
            "direct": false,
            "version": "1.5.1",
            "ecosystem": "crates"
          },
          {
            "name": "icu_properties_data",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_provider",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_provider",
            "direct": false,
            "version": "2.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "icu_provider_macros",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "ident_case",
            "direct": false,
            "version": "1.0.1",
            "ecosystem": "crates"
          },
          {
            "name": "idna",
            "direct": false,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "idna_adapter",
            "direct": false,
            "version": "1.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "indexmap",
            "direct": false,
            "version": "2.14.0",
            "ecosystem": "crates"
          },
          {
            "name": "indicatif",
            "direct": false,
            "version": "0.17.11",
            "ecosystem": "crates"
          },
          {
            "name": "indicatif",
            "direct": false,
            "version": "0.18.6",
            "ecosystem": "crates"
          },
          {
            "name": "is-terminal",
            "direct": false,
            "version": "0.4.17",
            "ecosystem": "crates"
          },
          {
            "name": "is_terminal_polyfill",
            "direct": false,
            "version": "1.70.2",
            "ecosystem": "crates"
          },
          {
            "name": "itertools",
            "direct": false,
            "version": "0.10.5",
            "ecosystem": "crates"
          },
          {
            "name": "itertools",
            "direct": false,
            "version": "0.14.0",
            "ecosystem": "crates"
          },
          {
            "name": "itoa",
            "direct": false,
            "version": "1.0.18",
            "ecosystem": "crates"
          },
          {
            "name": "js-sys",
            "direct": false,
            "version": "0.3.103",
            "ecosystem": "crates"
          },
          {
            "name": "lazy_static",
            "direct": false,
            "version": "1.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "libc",
            "direct": false,
            "version": null,
            "ecosystem": "crates"
          },
          {
            "name": "libc",
            "direct": false,
            "version": "0.2.189",
            "ecosystem": "crates"
          },
          {
            "name": "libredox",
            "direct": false,
            "version": "0.1.18",
            "ecosystem": "crates"
          },
          {
            "name": "litemap",
            "direct": false,
            "version": "0.7.5",
            "ecosystem": "crates"
          },
          {
            "name": "litemap",
            "direct": false,
            "version": "0.8.2",
            "ecosystem": "crates"
          },
          {
            "name": "lock_api",
            "direct": false,
            "version": "0.4.14",
            "ecosystem": "crates"
          },
          {
            "name": "log",
            "direct": false,
            "version": "0.4.33",
            "ecosystem": "crates"
          },
          {
            "name": "macro_rules_attribute",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "macro_rules_attribute-proc_macro",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "matrixmultiply",
            "direct": false,
            "version": "0.3.11",
            "ecosystem": "crates"
          },
          {
            "name": "minimal-lexical",
            "direct": false,
            "version": "0.2.1",
            "ecosystem": "crates"
          },
          {
            "name": "miniz_oxide",
            "direct": false,
            "version": "0.8.9",
            "ecosystem": "crates"
          },
          {
            "name": "monostate",
            "direct": false,
            "version": "0.1.18",
            "ecosystem": "crates"
          },
          {
            "name": "monostate-impl",
            "direct": false,
            "version": "0.1.18",
            "ecosystem": "crates"
          },
          {
            "name": "ndarray",
            "direct": false,
            "version": "0.17.2",
            "ecosystem": "crates"
          },
          {
            "name": "nom",
            "direct": false,
            "version": "7.1.3",
            "ecosystem": "crates"
          },
          {
            "name": "num-complex",
            "direct": false,
            "version": "0.4.6",
            "ecosystem": "crates"
          },
          {
            "name": "num-integer",
            "direct": false,
            "version": "0.1.46",
            "ecosystem": "crates"
          },
          {
            "name": "num-traits",
            "direct": false,
            "version": "0.2.19",
            "ecosystem": "crates"
          },
          {
            "name": "number_prefix",
            "direct": false,
            "version": "0.4.0",
            "ecosystem": "crates"
          },
          {
            "name": "once_cell",
            "direct": false,
            "version": "1.21.4",
            "ecosystem": "crates"
          },
          {
            "name": "once_cell_polyfill",
            "direct": false,
            "version": "1.70.2",
            "ecosystem": "crates"
          },
          {
            "name": "onig",
            "direct": false,
            "version": "6.5.3",
            "ecosystem": "crates"
          },
          {
            "name": "onig_sys",
            "direct": false,
            "version": "69.9.3",
            "ecosystem": "crates"
          },
          {
            "name": "oorandom",
            "direct": false,
            "version": "11.1.5",
            "ecosystem": "crates"
          },
          {
            "name": "option-ext",
            "direct": false,
            "version": "0.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "parking_lot",
            "direct": false,
            "version": "0.12.5",
            "ecosystem": "crates"
          },
          {
            "name": "parking_lot_core",
            "direct": false,
            "version": "0.9.12",
            "ecosystem": "crates"
          },
          {
            "name": "paste",
            "direct": false,
            "version": "1.0.15",
            "ecosystem": "crates"
          },
          {
            "name": "percent-encoding",
            "direct": false,
            "version": "2.3.2",
            "ecosystem": "crates"
          },
          {
            "name": "pin-project-lite",
            "direct": false,
            "version": "0.2.17",
            "ecosystem": "crates"
          },
          {
            "name": "pkg-config",
            "direct": false,
            "version": "0.3.33",
            "ecosystem": "crates"
          },
          {
            "name": "plotters",
            "direct": false,
            "version": "0.3.7",
            "ecosystem": "crates"
          },
          {
            "name": "plotters-backend",
            "direct": false,
            "version": "0.3.7",
            "ecosystem": "crates"
          },
          {
            "name": "plotters-svg",
            "direct": false,
            "version": "0.3.7",
            "ecosystem": "crates"
          },
          {
            "name": "portable-atomic",
            "direct": false,
            "version": "1.14.0",
            "ecosystem": "crates"
          },
          {
            "name": "portable-atomic-util",
            "direct": false,
            "version": "0.2.7",
            "ecosystem": "crates"
          },
          {
            "name": "potential_utf",
            "direct": false,
            "version": "0.1.5",
            "ecosystem": "crates"
          },
          {
            "name": "ppv-lite86",
            "direct": false,
            "version": "0.2.21",
            "ecosystem": "crates"
          },
          {
            "name": "proc-macro2",
            "direct": false,
            "version": "1.0.107",
            "ecosystem": "crates"
          },
          {
            "name": "pyo3-build-config",
            "direct": false,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "pyo3-ffi",
            "direct": false,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "pyo3-macros",
            "direct": false,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "pyo3-macros-backend",
            "direct": false,
            "version": "0.29.0",
            "ecosystem": "crates"
          },
          {
            "name": "quote",
            "direct": false,
            "version": "1.0.47",
            "ecosystem": "crates"
          },
          {
            "name": "r-efi",
            "direct": false,
            "version": "5.3.0",
            "ecosystem": "crates"
          },
          {
            "name": "rand",
            "direct": false,
            "version": "0.9.5",
            "ecosystem": "crates"
          },
          {
            "name": "rand_chacha",
            "direct": false,
            "version": "0.9.0",
            "ecosystem": "crates"
          },
          {
            "name": "rand_core",
            "direct": false,
            "version": "0.9.5",
            "ecosystem": "crates"
          },
          {
            "name": "rawpointer",
            "direct": false,
            "version": "0.2.1",
            "ecosystem": "crates"
          },
          {
            "name": "rayon",
            "direct": false,
            "version": "1.12.0",
            "ecosystem": "crates"
          },
          {
            "name": "rayon-cond",
            "direct": false,
            "version": "0.4.0",
            "ecosystem": "crates"
          },
          {
            "name": "rayon-core",
            "direct": false,
            "version": "1.13.0",
            "ecosystem": "crates"
          },
          {
            "name": "redox_syscall",
            "direct": false,
            "version": "0.5.18",
            "ecosystem": "crates"
          },
          {
            "name": "redox_users",
            "direct": false,
            "version": "0.4.6",
            "ecosystem": "crates"
          },
          {
            "name": "redox_users",
            "direct": false,
            "version": "0.5.2",
            "ecosystem": "crates"
          },
          {
            "name": "regex",
            "direct": false,
            "version": "1.13.1",
            "ecosystem": "crates"
          },
          {
            "name": "regex-syntax",
            "direct": false,
            "version": "0.8.11",
            "ecosystem": "crates"
          },
          {
            "name": "ring",
            "direct": false,
            "version": "0.17.14",
            "ecosystem": "crates"
          },
          {
            "name": "rustc-hash",
            "direct": false,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "rustc-hash",
            "direct": false,
            "version": "2.1.3",
            "ecosystem": "crates"
          },
          {
            "name": "rustls",
            "direct": false,
            "version": "0.23.42",
            "ecosystem": "crates"
          },
          {
            "name": "rustls-pki-types",
            "direct": false,
            "version": "1.15.1",
            "ecosystem": "crates"
          },
          {
            "name": "rustls-webpki",
            "direct": false,
            "version": "0.103.13",
            "ecosystem": "crates"
          },
          {
            "name": "rustversion",
            "direct": false,
            "version": "1.0.23",
            "ecosystem": "crates"
          },
          {
            "name": "ryu",
            "direct": false,
            "version": "1.0.23",
            "ecosystem": "crates"
          },
          {
            "name": "same-file",
            "direct": false,
            "version": "1.0.6",
            "ecosystem": "crates"
          },
          {
            "name": "scopeguard",
            "direct": false,
            "version": "1.2.0",
            "ecosystem": "crates"
          },
          {
            "name": "serde_core",
            "direct": false,
            "version": "1.0.229",
            "ecosystem": "crates"
          },
          {
            "name": "serde_derive",
            "direct": false,
            "version": "1.0.229",
            "ecosystem": "crates"
          },
          {
            "name": "shlex",
            "direct": false,
            "version": "2.0.1",
            "ecosystem": "crates"
          },
          {
            "name": "simd-adler32",
            "direct": false,
            "version": "0.3.10",
            "ecosystem": "crates"
          },
          {
            "name": "slab",
            "direct": false,
            "version": "0.4.12",
            "ecosystem": "crates"
          },
          {
            "name": "socks",
            "direct": false,
            "version": "0.3.4",
            "ecosystem": "crates"
          },
          {
            "name": "spm_precompiled",
            "direct": false,
            "version": "0.1.4",
            "ecosystem": "crates"
          },
          {
            "name": "stable_deref_trait",
            "direct": false,
            "version": "1.2.1",
            "ecosystem": "crates"
          },
          {
            "name": "static_assertions",
            "direct": false,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "strsim",
            "direct": false,
            "version": "0.11.1",
            "ecosystem": "crates"
          },
          {
            "name": "subtle",
            "direct": false,
            "version": "2.6.1",
            "ecosystem": "crates"
          },
          {
            "name": "syn",
            "direct": false,
            "version": "2.0.119",
            "ecosystem": "crates"
          },
          {
            "name": "syn",
            "direct": false,
            "version": "3.0.3",
            "ecosystem": "crates"
          },
          {
            "name": "synstructure",
            "direct": false,
            "version": "0.13.2",
            "ecosystem": "crates"
          },
          {
            "name": "target-lexicon",
            "direct": false,
            "version": "0.13.5",
            "ecosystem": "crates"
          },
          {
            "name": "thiserror",
            "direct": false,
            "version": "1.0.69",
            "ecosystem": "crates"
          },
          {
            "name": "thiserror",
            "direct": false,
            "version": "2.0.19",
            "ecosystem": "crates"
          },
          {
            "name": "thiserror-impl",
            "direct": false,
            "version": "1.0.69",
            "ecosystem": "crates"
          },
          {
            "name": "thiserror-impl",
            "direct": false,
            "version": "2.0.19",
            "ecosystem": "crates"
          },
          {
            "name": "tinystr",
            "direct": false,
            "version": "0.7.6",
            "ecosystem": "crates"
          },
          {
            "name": "tinystr",
            "direct": false,
            "version": "0.8.3",
            "ecosystem": "crates"
          },
          {
            "name": "tinytemplate",
            "direct": false,
            "version": "1.2.1",
            "ecosystem": "crates"
          },
          {
            "name": "unicode-ident",
            "direct": false,
            "version": "1.0.24",
            "ecosystem": "crates"
          },
          {
            "name": "unicode-normalization-alignments",
            "direct": false,
            "version": "0.1.12",
            "ecosystem": "crates"
          },
          {
            "name": "unicode-width",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "unicode_categories",
            "direct": false,
            "version": "0.1.1",
            "ecosystem": "crates"
          },
          {
            "name": "unit-prefix",
            "direct": false,
            "version": "0.5.2",
            "ecosystem": "crates"
          },
          {
            "name": "untrusted",
            "direct": false,
            "version": "0.9.0",
            "ecosystem": "crates"
          },
          {
            "name": "url",
            "direct": false,
            "version": "2.5.8",
            "ecosystem": "crates"
          },
          {
            "name": "utf16_iter",
            "direct": false,
            "version": "1.0.5",
            "ecosystem": "crates"
          },
          {
            "name": "utf8_iter",
            "direct": false,
            "version": "1.0.4",
            "ecosystem": "crates"
          },
          {
            "name": "utf8parse",
            "direct": false,
            "version": "0.2.2",
            "ecosystem": "crates"
          },
          {
            "name": "version_check",
            "direct": false,
            "version": "0.9.5",
            "ecosystem": "crates"
          },
          {
            "name": "walkdir",
            "direct": false,
            "version": "2.5.0",
            "ecosystem": "crates"
          },
          {
            "name": "wasi",
            "direct": false,
            "version": "0.11.1+wasi-snapshot-preview1",
            "ecosystem": "crates"
          },
          {
            "name": "wasip2",
            "direct": false,
            "version": "1.0.4+wasi-0.2.12",
            "ecosystem": "crates"
          },
          {
            "name": "wasm-bindgen",
            "direct": false,
            "version": "0.2.126",
            "ecosystem": "crates"
          },
          {
            "name": "wasm-bindgen-macro",
            "direct": false,
            "version": "0.2.126",
            "ecosystem": "crates"
          },
          {
            "name": "wasm-bindgen-macro-support",
            "direct": false,
            "version": "0.2.126",
            "ecosystem": "crates"
          },
          {
            "name": "wasm-bindgen-shared",
            "direct": false,
            "version": "0.2.126",
            "ecosystem": "crates"
          },
          {
            "name": "web-sys",
            "direct": false,
            "version": "0.3.103",
            "ecosystem": "crates"
          },
          {
            "name": "web-time",
            "direct": false,
            "version": "1.1.0",
            "ecosystem": "crates"
          },
          {
            "name": "webpki-roots",
            "direct": false,
            "version": "0.26.11",
            "ecosystem": "crates"
          },
          {
            "name": "webpki-roots",
            "direct": false,
            "version": "1.0.9",
            "ecosystem": "crates"
          },
          {
            "name": "winapi",
            "direct": false,
            "version": "0.3.9",
            "ecosystem": "crates"
          },
          {
            "name": "winapi-i686-pc-windows-gnu",
            "direct": false,
            "version": "0.4.0",
            "ecosystem": "crates"
          },
          {
            "name": "winapi-util",
            "direct": false,
            "version": "0.1.11",
            "ecosystem": "crates"
          },
          {
            "name": "winapi-x86_64-pc-windows-gnu",
            "direct": false,
            "version": "0.4.0",
            "ecosystem": "crates"
          },
          {
            "name": "windows-link",
            "direct": false,
            "version": "0.2.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows-sys",
            "direct": false,
            "version": "0.52.0",
            "ecosystem": "crates"
          },
          {
            "name": "windows-sys",
            "direct": false,
            "version": "0.59.0",
            "ecosystem": "crates"
          },
          {
            "name": "windows-sys",
            "direct": false,
            "version": "0.60.2",
            "ecosystem": "crates"
          },
          {
            "name": "windows-sys",
            "direct": false,
            "version": "0.61.2",
            "ecosystem": "crates"
          },
          {
            "name": "windows-targets",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows-targets",
            "direct": false,
            "version": "0.53.5",
            "ecosystem": "crates"
          },
          {
            "name": "windows_aarch64_gnullvm",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_aarch64_gnullvm",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_aarch64_msvc",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_aarch64_msvc",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_gnu",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_gnu",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_gnullvm",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_gnullvm",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_msvc",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_i686_msvc",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_gnu",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_gnu",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_gnullvm",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_gnullvm",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_msvc",
            "direct": false,
            "version": "0.52.6",
            "ecosystem": "crates"
          },
          {
            "name": "windows_x86_64_msvc",
            "direct": false,
            "version": "0.53.1",
            "ecosystem": "crates"
          },
          {
            "name": "wit-bindgen",
            "direct": false,
            "version": "0.57.1",
            "ecosystem": "crates"
          },
          {
            "name": "write16",
            "direct": false,
            "version": "1.0.0",
            "ecosystem": "crates"
          },
          {
            "name": "writeable",
            "direct": false,
            "version": "0.5.5",
            "ecosystem": "crates"
          },
          {
            "name": "writeable",
            "direct": false,
            "version": "0.6.3",
            "ecosystem": "crates"
          },
          {
            "name": "yoke",
            "direct": false,
            "version": "0.7.5",
            "ecosystem": "crates"
          },
          {
            "name": "yoke",
            "direct": false,
            "version": "0.8.3",
            "ecosystem": "crates"
          },
          {
            "name": "yoke-derive",
            "direct": false,
            "version": "0.7.5",
            "ecosystem": "crates"
          },
          {
            "name": "yoke-derive",
            "direct": false,
            "version": "0.8.2",
            "ecosystem": "crates"
          },
          {
            "name": "zerocopy",
            "direct": false,
            "version": "0.8.55",
            "ecosystem": "crates"
          },
          {
            "name": "zerocopy-derive",
            "direct": false,
            "version": "0.8.55",
            "ecosystem": "crates"
          },
          {
            "name": "zerofrom",
            "direct": false,
            "version": "0.1.8",
            "ecosystem": "crates"
          },
          {
            "name": "zerofrom-derive",
            "direct": false,
            "version": "0.1.7",
            "ecosystem": "crates"
          },
          {
            "name": "zeroize",
            "direct": false,
            "version": "1.9.0",
            "ecosystem": "crates"
          },
          {
            "name": "zerotrie",
            "direct": false,
            "version": "0.2.4",
            "ecosystem": "crates"
          },
          {
            "name": "zerovec",
            "direct": false,
            "version": "0.10.4",
            "ecosystem": "crates"
          },
          {
            "name": "zerovec",
            "direct": false,
            "version": "0.11.6",
            "ecosystem": "crates"
          },
          {
            "name": "zerovec-derive",
            "direct": false,
            "version": "0.10.3",
            "ecosystem": "crates"
          },
          {
            "name": "zerovec-derive",
            "direct": false,
            "version": "0.11.3",
            "ecosystem": "crates"
          },
          {
            "name": "zmij",
            "direct": false,
            "version": "1.0.23",
            "ecosystem": "crates"
          },
          {
            "name": "zopfli",
            "direct": false,
            "version": "0.8.3",
            "ecosystem": "crates"
          },
          {
            "name": "maturin",
            "direct": false,
            "version": null,
            "ecosystem": "pypi"
          }
        ],
        "collected": true,
        "truncated": false,
        "total_count": 307,
        "direct_count": 48,
        "indirect_count": 259
      }
    },
    "maintainership": {
      "issues": {
        "open_prs": 0,
        "merged_prs": 8,
        "open_issues": 3,
        "closed_ratio": 0.25,
        "closed_issues": 1,
        "closed_unmerged_prs": 0
      },
      "bus_factor": 1,
      "bot_contributors": 0,
      "top_contributors": [
        {
          "type": "User",
          "login": "chonknick",
          "commits": 168,
          "avatar_url": "https://avatars.githubusercontent.com/u/206151257?v=4"
        },
        {
          "type": "User",
          "login": "chonk-lain",
          "commits": 20,
          "avatar_url": "https://avatars.githubusercontent.com/u/224303370?v=4"
        }
      ],
      "contributors_sampled": 2,
      "top_contributor_share": 0.894
    },
    "quality_signals": {
      "has_ci": true,
      "has_tests": true,
      "ci_workflows": [
        "pyodide.yml",
        "release.yml",
        "speed-regression.yml",
        "tokenizer-accuracy.yml"
      ],
      "has_docs_dir": true,
      "linter_configs": [],
      "has_editorconfig": false,
      "has_linter_config": false,
      "has_precommit_config": false
    },
    "security_signals": {
      "lockfiles": [
        "Cargo.lock"
      ],
      "scorecard": {
        "checks": [
          {
            "name": "Binary-Artifacts",
            "score": 10,
            "reason": "no binaries found in the repo",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#binary-artifacts"
          },
          {
            "name": "Branch-Protection",
            "score": 1,
            "reason": "branch protection is not maximal on development and all release branches",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#branch-protection"
          },
          {
            "name": "CI-Tests",
            "score": null,
            "reason": "no pull request found",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#ci-tests"
          },
          {
            "name": "CII-Best-Practices",
            "score": 0,
            "reason": "no effort to earn an OpenSSF best practices badge detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#cii-best-practices"
          },
          {
            "name": "Code-Review",
            "score": 0,
            "reason": "Found 0/30 approved changesets -- score normalized to 0",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#code-review"
          },
          {
            "name": "Contributors",
            "score": 6,
            "reason": "project has 2 contributing companies or organizations -- score normalized to 6",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#contributors"
          },
          {
            "name": "Dangerous-Workflow",
            "score": 10,
            "reason": "no dangerous workflow patterns detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#dangerous-workflow"
          },
          {
            "name": "Dependency-Update-Tool",
            "score": 0,
            "reason": "no update tool detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#dependency-update-tool"
          },
          {
            "name": "Fuzzing",
            "score": 0,
            "reason": "project is not fuzzed",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#fuzzing"
          },
          {
            "name": "License",
            "score": 10,
            "reason": "license file detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#license"
          },
          {
            "name": "Maintained",
            "score": 10,
            "reason": "30 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#maintained"
          },
          {
            "name": "Packaging",
            "score": 10,
            "reason": "packaging workflow detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#packaging"
          },
          {
            "name": "Pinned-Dependencies",
            "score": 0,
            "reason": "dependency not pinned by hash detected -- score normalized to 0",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#pinned-dependencies"
          },
          {
            "name": "SAST",
            "score": 0,
            "reason": "no SAST tool detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#sast"
          },
          {
            "name": "Security-Policy",
            "score": 0,
            "reason": "security policy file not detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#security-policy"
          },
          {
            "name": "Signed-Releases",
            "score": null,
            "reason": "no releases found",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#signed-releases"
          },
          {
            "name": "Token-Permissions",
            "score": 0,
            "reason": "detected GitHub workflow tokens with excessive permissions",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#token-permissions"
          },
          {
            "name": "Vulnerabilities",
            "score": 8,
            "reason": "2 existing vulnerabilities detected",
            "documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#vulnerabilities"
          }
        ],
        "commit": "180d127346cf530c18b9229dfcdf4165bfe84e1f",
        "ran_at": "2026-07-24T12:09:02Z",
        "aggregate_score": 4.3,
        "scorecard_version": "v5.5.0"
      },
      "has_codeql_workflow": false,
      "has_security_policy": false,
      "has_dependabot_config": false
    },
    "contribution_flow": {
      "collected": true,
      "ci_last_run_at": "2026-07-24T09:58:13Z",
      "oldest_open_prs": [],
      "last_merged_pr_at": "2026-07-23T21:50:22Z",
      "ci_last_conclusion": "SUCCESS",
      "oldest_open_issues": [
        {
          "number": 6,
          "created_at": "2026-05-09T04:29:41Z",
          "last_comment_at": null,
          "last_comment_author": null
        },
        {
          "number": 11,
          "created_at": "2026-06-18T11:59:12Z",
          "last_comment_at": null,
          "last_comment_author": null
        },
        {
          "number": 12,
          "created_at": "2026-07-01T23:20:13Z",
          "last_comment_at": null,
          "last_comment_author": null
        }
      ]
    }
  },
  "config": {
    "disabled_metrics": [],
    "disabled_categories": [],
    "disabled_components": {}
  },
  "source": {
    "url": "https://github.com/feyninc/tokie",
    "host": "github.com",
    "name": "tokie",
    "owner": "feyninc"
  },
  "metrics": {
    "overall": {
      "key": "overall",
      "band": "moderate",
      "name": "Overall health",
      "note": null,
      "notes": [],
      "value": 57,
      "inputs": {
        "security": 54,
        "vitality": 83,
        "community": 40,
        "governance": 41,
        "engineering": 66
      },
      "components": []
    },
    "categories": [
      {
        "key": "vitality",
        "band": "good",
        "name": "Vitality",
        "value": 83,
        "weight": 0.22,
        "metrics": [
          {
            "key": "development_activity",
            "band": "good",
            "name": "Development activity",
            "note": null,
            "notes": [],
            "value": 71,
            "inputs": {
              "commits_last_year": 168,
              "human_commit_share": 1,
              "days_since_last_push": 0,
              "active_weeks_last_year": 10
            },
            "components": [
              {
                "key": "push_recency",
                "name": "Push recency",
                "detail": "last push 0 days ago",
                "points": 36,
                "status": "met",
                "details": [
                  {
                    "code": "push_recency",
                    "params": {
                      "days": 0
                    }
                  }
                ],
                "max_points": 36
              },
              {
                "key": "commit_cadence",
                "name": "Commit cadence",
                "detail": "10/52 weeks with commits",
                "points": 6.9,
                "status": "partial",
                "details": [
                  {
                    "code": "commit_cadence_weeks",
                    "params": {
                      "weeks": 10
                    }
                  }
                ],
                "max_points": 36
              },
              {
                "key": "commit_volume",
                "name": "Commit volume",
                "detail": "168 commits in the last year",
                "points": 18,
                "status": "met",
                "details": [
                  {
                    "code": "commits_last_year",
                    "params": {
                      "count": 168
                    }
                  }
                ],
                "max_points": 18
              },
              {
                "key": "openssf_scorecard_maintained",
                "name": "OpenSSF Scorecard: Maintained",
                "detail": "30 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10",
                "points": 10,
                "status": "met",
                "details": [],
                "max_points": 10
              }
            ]
          },
          {
            "key": "release_discipline",
            "band": "excellent",
            "name": "Release discipline",
            "note": "Excluded from scoring (no data or not applicable): OpenSSF Scorecard: Signed-Releases. Remaining weights renormalized.",
            "notes": [
              {
                "code": "excluded_no_data",
                "params": {
                  "components": [
                    "openssf_scorecard_signed_releases"
                  ]
                }
              },
              {
                "code": "weights_renormalized",
                "params": {}
              }
            ],
            "value": 100,
            "inputs": {
              "releases_count": 14,
              "latest_release_tag": "v0.1.3",
              "releases_from_tags": false,
              "days_since_latest_release": 0,
              "mean_days_between_releases": 13.4
            },
            "components": [
              {
                "key": "ships_releases",
                "name": "Ships releases",
                "detail": "14 releases published",
                "points": 27,
                "status": "met",
                "details": [
                  {
                    "code": "releases_published",
                    "params": {
                      "count": 14
                    }
                  }
                ],
                "max_points": 27
              },
              {
                "key": "release_recency",
                "name": "Release recency",
                "detail": "latest release 0 days ago",
                "points": 36,
                "status": "met",
                "details": [
                  {
                    "code": "release_recency",
                    "params": {
                      "days": 0
                    }
                  }
                ],
                "max_points": 36
              },
              {
                "key": "release_cadence",
                "name": "Release cadence",
                "detail": "a release every ~13.4 days",
                "points": 27,
                "status": "met",
                "details": [
                  {
                    "code": "release_cadence",
                    "params": {
                      "gap": 13.4
                    }
                  }
                ],
                "max_points": 27
              },
              {
                "key": "openssf_scorecard_signed_releases",
                "name": "OpenSSF Scorecard: Signed-Releases",
                "detail": "no releases found",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "no_data",
                    "params": {}
                  }
                ],
                "max_points": 10
              }
            ]
          },
          {
            "key": "abandonment",
            "band": "excellent",
            "name": "Abandonment",
            "note": null,
            "notes": [],
            "value": 100,
            "inputs": {
              "cap": null,
              "state": "maintained",
              "guards": [],
              "signals": [],
              "red_flag": false,
              "multiplier_pct": 100,
              "declared_reason": null,
              "unverified_reason": null,
              "unanswered_open_prs": null,
              "unanswered_open_issues": null,
              "days_since_last_merged_pr": null,
              "days_since_last_human_commit": 0,
              "days_since_last_human_commit_is_floor": false
            },
            "components": [
              {
                "key": "project_is_still_maintained",
                "name": "Project is still maintained",
                "detail": "last human commit 0 days ago",
                "points": 100,
                "status": "met",
                "details": [
                  {
                    "code": "abandonment_maintained",
                    "params": {
                      "days": 0
                    }
                  }
                ],
                "max_points": 100
              }
            ]
          }
        ],
        "description": "Is the project alive — is code being written and are releases shipping?"
      },
      {
        "key": "community",
        "band": "at_risk",
        "name": "Community & Adoption",
        "value": 40,
        "weight": 0.18,
        "metrics": [
          {
            "key": "popularity",
            "band": "at_risk",
            "name": "Popularity & adoption",
            "note": null,
            "notes": [],
            "value": 32,
            "inputs": {
              "forks": 5,
              "stars": 49,
              "watchers": 0,
              "growth_state": "unverified",
              "growth_factor_pct": 100,
              "growth_unverified_reason": "no_history"
            },
            "components": [
              {
                "key": "stars",
                "name": "Stars",
                "detail": "49 stars",
                "points": 27.3,
                "status": "partial",
                "details": [
                  {
                    "code": "stars",
                    "params": {
                      "count": 49
                    }
                  }
                ],
                "max_points": 60
              },
              {
                "key": "forks",
                "name": "Forks",
                "detail": "5 forks",
                "points": 5,
                "status": "partial",
                "details": [
                  {
                    "code": "forks",
                    "params": {
                      "count": 5
                    }
                  }
                ],
                "max_points": 25
              },
              {
                "key": "watchers",
                "name": "Watchers",
                "detail": "0 watchers",
                "points": 0,
                "status": "missed",
                "details": [
                  {
                    "code": "watchers",
                    "params": {
                      "count": 0
                    }
                  }
                ],
                "max_points": 15
              }
            ]
          },
          {
            "key": "community_health",
            "band": "moderate",
            "name": "Community health",
            "note": null,
            "notes": [],
            "value": 50,
            "inputs": {
              "has_readme": true,
              "has_license": true,
              "has_contributing": false,
              "has_issue_template": false,
              "has_code_of_conduct": false,
              "has_pull_request_template": false
            },
            "components": [
              {
                "key": "readme",
                "name": "README",
                "detail": null,
                "points": 22.5,
                "status": "met",
                "details": [],
                "max_points": 22.5
              },
              {
                "key": "license",
                "name": "License",
                "detail": "recognized license (Apache-2.0)",
                "points": 22.5,
                "status": "met",
                "details": [
                  {
                    "code": "license_standard",
                    "params": {}
                  },
                  {
                    "code": "license_spdx",
                    "params": {
                      "spdx": "Apache-2.0"
                    }
                  }
                ],
                "max_points": 22.5
              },
              {
                "key": "contributing_guide",
                "name": "CONTRIBUTING guide",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 18
              },
              {
                "key": "code_of_conduct",
                "name": "Code of conduct",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 13.5
              },
              {
                "key": "issue_template",
                "name": "Issue template",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 7.2
              },
              {
                "key": "pr_template",
                "name": "PR template",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 6.3
              }
            ]
          }
        ],
        "description": "Does the project have users, downloads, attention, and a welcoming setup for contributors?"
      },
      {
        "key": "governance",
        "band": "at_risk",
        "name": "Sustainability & Governance",
        "value": 41,
        "weight": 0.24,
        "metrics": [
          {
            "key": "maintainer_resilience",
            "band": "critical",
            "name": "Maintainer resilience (bus factor)",
            "note": null,
            "notes": [],
            "value": 20,
            "inputs": {
              "bus_factor": 1,
              "contributors_sampled": 2,
              "top_contributor_share": 0.894
            },
            "components": [
              {
                "key": "bus_factor",
                "name": "Bus factor",
                "detail": "1 contributor(s) cover half of all commits",
                "points": 9,
                "status": "partial",
                "details": [
                  {
                    "code": "bus_factor",
                    "params": {
                      "count": 1
                    }
                  }
                ],
                "max_points": 54
              },
              {
                "key": "commit_distribution",
                "name": "Commit distribution",
                "detail": "top contributor authored 89% of commits",
                "points": 2.4,
                "status": "partial",
                "details": [
                  {
                    "code": "top_contributor_share",
                    "params": {
                      "share": 89
                    }
                  }
                ],
                "max_points": 22.5
              },
              {
                "key": "contributor_breadth",
                "name": "Contributor breadth",
                "detail": "2 contributors",
                "points": 2.7,
                "status": "partial",
                "details": [
                  {
                    "code": "contributors_sampled",
                    "params": {
                      "count": 2
                    }
                  }
                ],
                "max_points": 13.5
              },
              {
                "key": "openssf_scorecard_contributors",
                "name": "OpenSSF Scorecard: Contributors",
                "detail": "project has 2 contributing companies or organizations -- score normalized to 6",
                "points": 6,
                "status": "partial",
                "details": [],
                "max_points": 10
              }
            ]
          },
          {
            "key": "responsiveness",
            "band": "moderate",
            "name": "Issue & PR responsiveness",
            "note": null,
            "notes": [],
            "value": 50,
            "inputs": {
              "merged_prs": 8,
              "open_issues": 3,
              "closed_issues": 1,
              "issue_closed_ratio": 0.25,
              "closed_unmerged_prs": 0
            },
            "components": [
              {
                "key": "issue_resolution",
                "name": "Issue resolution",
                "detail": "25% of issues closed",
                "points": 11.7,
                "status": "partial",
                "details": [
                  {
                    "code": "issues_closed_share",
                    "params": {
                      "share": 25
                    }
                  }
                ],
                "max_points": 46.75
              },
              {
                "key": "pr_acceptance",
                "name": "PR acceptance",
                "detail": "8/8 decided PRs merged",
                "points": 38.2,
                "status": "met",
                "details": [
                  {
                    "code": "decided_prs_merged",
                    "params": {
                      "merged": 8,
                      "decided": 8
                    }
                  }
                ],
                "max_points": 38.25
              },
              {
                "key": "openssf_scorecard_code_review",
                "name": "OpenSSF Scorecard: Code-Review",
                "detail": "Found 0/30 approved changesets -- score normalized to 0",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 15
              }
            ]
          },
          {
            "key": "stewardship",
            "band": "moderate",
            "name": "Ownership & stewardship",
            "note": null,
            "notes": [],
            "value": 57,
            "inputs": {
              "followers": 150,
              "owner_type": "Organization",
              "is_verified": null,
              "owner_login": "feyninc",
              "public_repos": 16,
              "account_age_days": 483
            },
            "components": [
              {
                "key": "ownership_backing",
                "name": "Ownership backing",
                "detail": "organization-owned",
                "points": 30,
                "status": "met",
                "details": [
                  {
                    "code": "owner_organization",
                    "params": {}
                  }
                ],
                "max_points": 30
              },
              {
                "key": "verified_domain",
                "name": "Verified domain",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 20
              },
              {
                "key": "owner_reach",
                "name": "Owner reach",
                "detail": "150 followers of feyninc",
                "points": 15.7,
                "status": "partial",
                "details": [
                  {
                    "code": "owner_followers",
                    "params": {
                      "count": 150,
                      "login": "feyninc"
                    }
                  }
                ],
                "max_points": 25
              },
              {
                "key": "track_record",
                "name": "Track record",
                "detail": "16 public repos, account ~1 yr old",
                "points": 11.6,
                "status": "partial",
                "details": [
                  {
                    "code": "public_repos",
                    "params": {
                      "count": 16
                    }
                  },
                  {
                    "code": "account_age_years",
                    "params": {
                      "years": 1
                    }
                  }
                ],
                "max_points": 25
              }
            ]
          }
        ],
        "description": "Will the project survive its people — bus factor, responsiveness, who backs it, and package upkeep?"
      },
      {
        "key": "engineering",
        "band": "moderate",
        "name": "Engineering Quality",
        "value": 66,
        "weight": 0.2,
        "metrics": [
          {
            "key": "engineering_practices",
            "band": "moderate",
            "name": "Engineering practices",
            "note": "Excluded from scoring (no data or not applicable): OpenSSF Scorecard: CI-Tests. Remaining weights renormalized.",
            "notes": [
              {
                "code": "excluded_no_data",
                "params": {
                  "components": [
                    "openssf_scorecard_ci_tests"
                  ]
                }
              },
              {
                "code": "weights_renormalized",
                "params": {}
              }
            ],
            "value": 60,
            "inputs": {
              "has_ci": true,
              "has_tests": true,
              "has_editorconfig": false,
              "has_linter_config": false,
              "has_precommit_config": false
            },
            "components": [
              {
                "key": "ci_workflows",
                "name": "CI workflows",
                "detail": "4 workflow(s)",
                "points": 24,
                "status": "met",
                "details": [
                  {
                    "code": "ci_workflows",
                    "params": {
                      "count": 4
                    }
                  }
                ],
                "max_points": 24
              },
              {
                "key": "tests_present",
                "name": "Tests present",
                "detail": null,
                "points": 24,
                "status": "met",
                "details": [],
                "max_points": 24
              },
              {
                "key": "linter_config",
                "name": "Linter config",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 16
              },
              {
                "key": "pre_commit_hooks",
                "name": "Pre-commit hooks",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 9.6
              },
              {
                "key": "editorconfig",
                "name": ".editorconfig",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 6.4
              },
              {
                "key": "openssf_scorecard_ci_tests",
                "name": "OpenSSF Scorecard: CI-Tests",
                "detail": "no pull request found",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "no_data",
                    "params": {}
                  }
                ],
                "max_points": 20
              }
            ]
          },
          {
            "key": "documentation",
            "band": "good",
            "name": "Documentation",
            "note": null,
            "notes": [],
            "value": 75,
            "inputs": {
              "topics": [],
              "has_wiki": true,
              "homepage": null,
              "has_readme": true,
              "has_docs_dir": true,
              "has_description": true
            },
            "components": [
              {
                "key": "readme",
                "name": "README",
                "detail": null,
                "points": 30,
                "status": "met",
                "details": [],
                "max_points": 30
              },
              {
                "key": "documentation_directory",
                "name": "Documentation directory",
                "detail": null,
                "points": 25,
                "status": "met",
                "details": [],
                "max_points": 25
              },
              {
                "key": "documentation_homepage_site",
                "name": "Documentation / homepage site",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 15
              },
              {
                "key": "repository_description",
                "name": "Repository description",
                "detail": null,
                "points": 10,
                "status": "met",
                "details": [],
                "max_points": 10
              },
              {
                "key": "topics",
                "name": "Topics",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 10
              },
              {
                "key": "wiki",
                "name": "Wiki",
                "detail": null,
                "points": 10,
                "status": "met",
                "details": [],
                "max_points": 10
              }
            ]
          }
        ],
        "description": "Are baseline engineering and documentation practices in place?"
      },
      {
        "key": "security",
        "band": "moderate",
        "name": "Security",
        "value": 54,
        "weight": 0.16,
        "metrics": [
          {
            "key": "security_posture",
            "band": "at_risk",
            "name": "Security posture",
            "note": "Excluded from scoring (no data or not applicable): CI-Tests, Signed-Releases. Remaining weights renormalized.",
            "notes": [
              {
                "code": "excluded_no_data",
                "params": {
                  "components": [
                    "ci_tests",
                    "signed_releases"
                  ]
                }
              },
              {
                "code": "weights_renormalized",
                "params": {}
              }
            ],
            "value": 43,
            "inputs": {
              "source": "openssf_scorecard",
              "checks_evaluated": 16,
              "scorecard_version": "v5.5.0",
              "checks_inconclusive": 2,
              "scorecard_aggregate": 4.3
            },
            "components": [
              {
                "key": "binary_artifacts",
                "name": "Binary-Artifacts",
                "detail": "no binaries found in the repo",
                "points": 7.5,
                "status": "met",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "branch_protection",
                "name": "Branch-Protection",
                "detail": "branch protection is not maximal on development and all release branches",
                "points": 0.8,
                "status": "partial",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "ci_tests",
                "name": "CI-Tests",
                "detail": "no pull request found",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "no_data",
                    "params": {}
                  }
                ],
                "max_points": 2.5
              },
              {
                "key": "cii_best_practices",
                "name": "CII-Best-Practices",
                "detail": "no effort to earn an OpenSSF best practices badge detected",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 2.5
              },
              {
                "key": "code_review",
                "name": "Code-Review",
                "detail": "Found 0/30 approved changesets -- score normalized to 0",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "contributors",
                "name": "Contributors",
                "detail": "project has 2 contributing companies or organizations -- score normalized to 6",
                "points": 1.5,
                "status": "partial",
                "details": [],
                "max_points": 2.5
              },
              {
                "key": "dangerous_workflow",
                "name": "Dangerous-Workflow",
                "detail": "no dangerous workflow patterns detected",
                "points": 10,
                "status": "met",
                "details": [],
                "max_points": 10
              },
              {
                "key": "dependency_update_tool",
                "name": "Dependency-Update-Tool",
                "detail": "no update tool detected",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "fuzzing",
                "name": "Fuzzing",
                "detail": "project is not fuzzed",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 5
              },
              {
                "key": "license",
                "name": "License",
                "detail": "license file detected",
                "points": 2.5,
                "status": "met",
                "details": [],
                "max_points": 2.5
              },
              {
                "key": "maintained",
                "name": "Maintained",
                "detail": "30 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10",
                "points": 7.5,
                "status": "met",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "packaging",
                "name": "Packaging",
                "detail": "packaging workflow detected",
                "points": 5,
                "status": "met",
                "details": [],
                "max_points": 5
              },
              {
                "key": "pinned_dependencies",
                "name": "Pinned-Dependencies",
                "detail": "dependency not pinned by hash detected -- score normalized to 0",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 5
              },
              {
                "key": "sast",
                "name": "SAST",
                "detail": "no SAST tool detected",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 5
              },
              {
                "key": "security_policy",
                "name": "Security-Policy",
                "detail": "security policy file not detected",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 5
              },
              {
                "key": "signed_releases",
                "name": "Signed-Releases",
                "detail": "no releases found",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "no_data",
                    "params": {}
                  }
                ],
                "max_points": 7.5
              },
              {
                "key": "token_permissions",
                "name": "Token-Permissions",
                "detail": "detected GitHub workflow tokens with excessive permissions",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 7.5
              },
              {
                "key": "vulnerabilities",
                "name": "Vulnerabilities",
                "detail": "2 existing vulnerabilities detected",
                "points": 6,
                "status": "partial",
                "details": [],
                "max_points": 7.5
              }
            ]
          },
          {
            "key": "dependency_advisories",
            "band": "excellent",
            "name": "Dependency advisories",
            "note": "Excluded from scoring (no data or not applicable): Indirect dependencies free of known advisories, No advisories left outstanding. Remaining weights renormalized. Matched 281 resolved dependencies against OSV; 26 could not be assessed (no resolved version, an unsupported ecosystem, or beyond the reported package list). This repository publishes no package the index resolves, so the repository dependency graph was assessed instead. That graph mixes development and test pins with shipped dependencies, so only the declared runtime dependencies are scored; transitive findings are reported as context and excluded from the score. Reachability is not analyzed.",
            "notes": [
              {
                "code": "excluded_no_data",
                "params": {
                  "components": [
                    "indirect_dependencies_free_of_known_advisories",
                    "no_advisories_left_outstanding"
                  ]
                }
              },
              {
                "code": "weights_renormalized",
                "params": {}
              },
              {
                "code": "advisories_scope_repository",
                "params": {
                  "assessed": 281
                }
              },
              {
                "code": "advisories_unassessed",
                "params": {
                  "count": 26
                }
              },
              {
                "code": "advisories_repo_graph_caveat",
                "params": {}
              },
              {
                "code": "advisories_reachability",
                "params": {}
              }
            ],
            "value": 100,
            "inputs": {
              "source": "osv",
              "advisories": 2,
              "affected_packages": 2,
              "assessed_packages": 281,
              "unassessed_packages": 26,
              "affected_by_severity": "unknown 2",
              "direct_affected_packages": 0
            },
            "components": [
              {
                "key": "direct_dependencies_free_of_known_advisories",
                "name": "Direct dependencies free of known advisories",
                "detail": "no direct dependency carries a known advisory",
                "points": 35,
                "status": "met",
                "details": [
                  {
                    "code": "no_direct_advisories",
                    "params": {}
                  }
                ],
                "max_points": 35
              },
              {
                "key": "indirect_dependencies_free_of_known_advisories",
                "name": "Indirect dependencies free of known advisories",
                "detail": "transitive set not separable from development and test dependencies in this scope",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "advisories_scope_not_separable",
                    "params": {}
                  }
                ],
                "max_points": 25
              },
              {
                "key": "no_advisories_left_outstanding",
                "name": "No advisories left outstanding",
                "detail": "no advisory carries a publication date",
                "points": 0,
                "status": "excluded",
                "details": [
                  {
                    "code": "advisories_no_publication_date",
                    "params": {}
                  }
                ],
                "max_points": 40
              }
            ]
          },
          {
            "key": "malicious_dependencies",
            "band": "excellent",
            "name": "Malicious dependencies",
            "note": null,
            "notes": [],
            "value": 100,
            "inputs": {
              "source": "osv",
              "meaning": "reported as a malicious package by the OpenSSF corpus; the remedy is removal or moving off the compromised name, never an upgrade of the same artifact. Versions the registry has since pulled are listed but not scored",
              "packages": [],
              "red_flag": false,
              "assessed_packages": 281,
              "malicious_packages": 0,
              "direct_malicious_packages": 0,
              "withdrawn_malicious_packages": 0,
              "installable_malicious_packages": 0
            },
            "components": [
              {
                "key": "no_dependency_reported_as_a_malicious_package",
                "name": "No dependency reported as a malicious package",
                "detail": "no dependency is reported as a malicious package",
                "points": 100,
                "status": "met",
                "details": [
                  {
                    "code": "no_malicious_dependencies",
                    "params": {}
                  }
                ],
                "max_points": 100
              }
            ]
          },
          {
            "key": "high_risk_jurisdiction_exposure",
            "band": "excellent",
            "name": "High-Risk Jurisdiction Exposure",
            "note": "Only high-confidence self-published location evidence affects this multiplier. Ambiguous matches are review-only; country evidence is not proof of nationality, citizenship, legal registration, malicious intent, or sanctions status.",
            "notes": [
              {
                "code": "jurisdiction_evidence_limits",
                "params": {}
              }
            ],
            "value": 100,
            "inputs": {
              "meaning": "self-published location evidence; not nationality or citizenship",
              "red_flag": false,
              "exposures": [],
              "policy_countries": [
                "Russia",
                "Iran",
                "North Korea"
              ],
              "review_only_matches": 0,
              "assessed_self_published_locations": 2
            },
            "components": [
              {
                "key": "policy_exposure_multiplier",
                "name": "Policy exposure multiplier",
                "detail": "no confirmed policy-scope location match",
                "points": 100,
                "status": "met",
                "details": [
                  {
                    "code": "jurisdiction_no_match",
                    "params": {}
                  }
                ],
                "max_points": 100
              }
            ]
          }
        ],
        "description": "Are visible security and supply-chain practices strong, with no malicious dependency and no unresolved high-risk jurisdiction exposure?"
      },
      {
        "key": "ai_readiness",
        "band": "moderate",
        "name": "AI Readiness",
        "value": 59,
        "weight": 0,
        "metrics": [
          {
            "key": "ai_agent_context",
            "band": "at_risk",
            "name": "Agent context & guidance",
            "note": null,
            "notes": [],
            "value": 40,
            "inputs": {
              "has_llms_txt": false,
              "legible_history_share": 0.77,
              "agent_instruction_files": [],
              "agent_instruction_max_bytes": null
            },
            "components": [
              {
                "key": "agent_instructions",
                "name": "Agent instructions",
                "detail": "no CLAUDE.md / AGENTS.md / editor rules",
                "points": 0,
                "status": "missed",
                "details": [
                  {
                    "code": "no_agent_instructions",
                    "params": {}
                  }
                ],
                "max_points": 45
              },
              {
                "key": "machine_readable_docs_llms_txt",
                "name": "Machine-readable docs (llms.txt)",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 15
              },
              {
                "key": "legible_commit_history",
                "name": "Legible commit history",
                "detail": "77 of 100 human commits state their intent (structured subject or explanatory body)",
                "points": 40,
                "status": "met",
                "details": [
                  {
                    "code": "legible_history",
                    "params": {
                      "legible": 77,
                      "sampled": 100
                    }
                  }
                ],
                "max_points": 40
              }
            ]
          },
          {
            "key": "ai_verify_loop",
            "band": "moderate",
            "name": "Verify loop (build / test / typecheck)",
            "note": null,
            "notes": [],
            "value": 66,
            "inputs": {
              "has_nix": false,
              "has_tests": true,
              "lockfiles": [
                "Cargo.lock"
              ],
              "has_dockerfile": false,
              "typed_language": true,
              "bootstrap_files": [],
              "has_devcontainer": false,
              "has_linter_config": false,
              "typecheck_configs": [],
              "agent_commit_share": 0.1,
              "toolchain_manifests": [
                "Cargo.toml",
                "crates/pretokie/Cargo.toml",
                "crates/tokie-cli/Cargo.toml",
                "crates/tokie-python/Cargo.toml",
                "crates/tokie/Cargo.toml"
              ],
              "dependency_bot_commit_share": 0
            },
            "components": [
              {
                "key": "one_command_bootstrap",
                "name": "One-command bootstrap",
                "detail": "Cargo.toml, crates/pretokie/Cargo.toml, crates/tokie-cli/Cargo.toml (toolchain convention, no task runner)",
                "points": 12.6,
                "status": "partial",
                "details": [
                  {
                    "code": "toolchain_convention",
                    "params": {
                      "files": "Cargo.toml, crates/pretokie/Cargo.toml, crates/tokie-cli/Cargo.toml"
                    }
                  }
                ],
                "max_points": 18
              },
              {
                "key": "automated_tests",
                "name": "Automated tests",
                "detail": null,
                "points": 22,
                "status": "met",
                "details": [],
                "max_points": 22
              },
              {
                "key": "lint_format_config",
                "name": "Lint / format config",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 11
              },
              {
                "key": "static_type_checking",
                "name": "Static type checking",
                "detail": "Rust (statically typed)",
                "points": 11,
                "status": "met",
                "details": [
                  {
                    "code": "statically_typed_language",
                    "params": {
                      "language": "Rust"
                    }
                  }
                ],
                "max_points": 11
              },
              {
                "key": "reproducible_environment",
                "name": "Reproducible environment",
                "detail": "lockfile",
                "points": 10,
                "status": "met",
                "details": [
                  {
                    "code": "file_list",
                    "params": {
                      "files": "lockfile"
                    }
                  }
                ],
                "max_points": 10
              },
              {
                "key": "demonstrated_agent_practice",
                "name": "Demonstrated agent practice",
                "detail": "10 of the last 100 commits agent-authored or agent-credited",
                "points": 10,
                "status": "met",
                "details": [
                  {
                    "code": "agent_authored_commits",
                    "params": {
                      "count": 10,
                      "sampled": 100
                    }
                  }
                ],
                "max_points": 10
              },
              {
                "key": "automated_maintenance",
                "name": "Automated maintenance",
                "detail": "no automated dependency updates observed",
                "points": 0,
                "status": "missed",
                "details": [
                  {
                    "code": "no_dependency_automation",
                    "params": {}
                  }
                ],
                "max_points": 8
              },
              {
                "key": "openssf_scorecard_pinned_dependencies",
                "name": "OpenSSF Scorecard: Pinned-Dependencies",
                "detail": "dependency not pinned by hash detected -- score normalized to 0",
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 10
              }
            ]
          },
          {
            "key": "ai_code_legibility",
            "band": "excellent",
            "name": "Code legibility for models",
            "note": null,
            "notes": [],
            "value": 98,
            "inputs": {
              "primary_language": "Rust",
              "largest_source_bytes": 90007,
              "source_files_sampled": 96,
              "oversized_source_files": 4
            },
            "components": [
              {
                "key": "type_checkable_code",
                "name": "Type-checkable code",
                "detail": "Rust (statically typed)",
                "points": 45,
                "status": "met",
                "details": [
                  {
                    "code": "statically_typed_language",
                    "params": {
                      "language": "Rust"
                    }
                  }
                ],
                "max_points": 45
              },
              {
                "key": "manageable_file_sizes",
                "name": "Manageable file sizes",
                "detail": "4/96 source files over 60KB",
                "points": 52.7,
                "status": "partial",
                "details": [
                  {
                    "code": "oversized_source_files",
                    "params": {
                      "kb": 60,
                      "sampled": 96,
                      "oversized": 4
                    }
                  }
                ],
                "max_points": 55
              }
            ]
          },
          {
            "key": "ai_interfaces",
            "band": "at_risk",
            "name": "Machine-readable interfaces",
            "note": null,
            "notes": [],
            "value": 40,
            "inputs": {
              "example_dirs": [
                "examples"
              ],
              "has_mcp_signal": false,
              "api_schema_files": []
            },
            "components": [
              {
                "key": "api_schema_openapi_graphql_proto",
                "name": "API schema (OpenAPI/GraphQL/proto)",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 40
              },
              {
                "key": "mcp_server",
                "name": "MCP server",
                "detail": null,
                "points": 0,
                "status": "missed",
                "details": [],
                "max_points": 20
              },
              {
                "key": "runnable_examples",
                "name": "Runnable examples",
                "detail": "examples",
                "points": 40,
                "status": "met",
                "details": [
                  {
                    "code": "file_list",
                    "params": {
                      "files": "examples"
                    }
                  }
                ],
                "max_points": 40
              }
            ]
          }
        ],
        "description": "How well is the repo equipped to be developed and maintained with AI coding agents? An independent, experimental badge — weight 0.0, so it is surfaced on its own and does not affect the overall health score."
      }
    ],
    "metrics_version": "1.13.0"
  },
  "warnings": [
    "Star history unavailable: GitHub GraphQL error: Resource not accessible by personal access token",
    "crates package 'tokie' points at a different repository (https://github.com/chonkie-inc/tokie); excluded from ecosystem scoring",
    "crates package 'pretokie' points at a different repository (https://github.com/chonkie-inc/tokie); excluded from ecosystem scoring",
    "Could not fetch crates package 'tokie-cli' from its registry",
    "Could not fetch crates package 'tokie-python' from its registry",
    "pypi package 'tokie' points at a different repository (https://github.com/chonkie-inc/tokie); excluded from ecosystem scoring",
    "deps.dev does not index crates:tokie@0.1.3; advisories assessed against the repository dependency graph instead"
  ],
  "report_type": "repository",
  "generated_at": "2026-07-24T12:09:09.239156Z",
  "schema_version": "0.27.0",
  "badge_url": "https://raw.githubusercontent.com/inspect-software/badges/main/v1/f/feyninc/tokie.svg",
  "full_name": "feyninc/tokie",
  "license_state": "standard",
  "license_spdx": "Apache-2.0"
}