Public recordAt risk
Software health reportschema 0.27.0 · metrics 1.13.0 · 2026-07-28 00:12 UTC
awee-ai / go-tokenizer
[WIP] Pure Go implementation of OpenAI's and Ollama, Anthropic tokenizers
awee-ai/go-tokenizer holds a health index of 32 out of 100, placing it in the At risk band. It scores highest on Vitality (55/100) and lowest on Community & Adoption (12/100). It was last updated 7 days ago. A single contributor accounts for most of its recent work.
32
overall / 100Software health index
Metrics are grouped into weighted categories on one standardized 1–100 scale. Overall starts as their weighted mean; when public evidence triggers the High-Risk Jurisdiction Policy, the rating is adjusted and receives an At risk ceiling of 49. AI Readiness sits outside the overall score.
32
Excellent85-100Exemplary; meets essentially all checked criteria
Good70-84Healthy; minor gaps
Moderate50-69Acceptable with notable gaps; review recommended
At risk30-49Significant weaknesses; adoption warrants caution
Critical1-29Severe problems (abandoned, single-maintainer, no hygiene)
Score profile
Each axis is a category. The shape matters more than the average — a healthy subject fills the whole shape, while a spike-and-crater profile means strength in one dimension is masking risk in another.
Ownership
AweeAIOrganization
0 followers10 public repossince Oct 2024
This repository is backed by an organization — shared, accountable stewardship that can outlive any single maintainer.
Package ecosystems
| Registry | Package | Version | Downloads / mo | Versions | Last publish |
|---|---|---|---|---|---|
| Go | github.com/awee-ai/go-tokenizer | v0.1.0 | - | 1 | 7 days ago |
Metrics by category
Vitality
Is the project alive — is code being written and are releases shipping?
55
Development activity
44At risk
How it's scored
| 36/36 | Push recency — last push 7 days ago |
| 1.4/36 | Commit cadence — 2/52 weeks with commits |
| 6.3/18 | Commit volume — 4 commits in the last year |
| 0/10 | OpenSSF Scorecard: Maintained — 1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 |
Inputs used
| commits_last_year | 4 |
| human_commit_share | 1 |
| days_since_last_push | 7 |
| active_weeks_last_year | 2 |
Release discipline
72Good
How it's scored
| 16.2/27 | Ships releases — 1 version tags (no GitHub releases) |
| 36/36 | Release recency — latest release 7 days ago |
| 12.6/27 | Release cadence — cadence unknown (single release) |
| 0/10 | OpenSSF Scorecard: Signed-Releases — no data |
Inputs used
| releases_count | 1 |
| latest_release_tag | v0.1.0 |
| releases_from_tags | yes |
| days_since_latest_release | 7 |
| mean_days_between_releases | — |
Excluded from scoring (no data or not applicable): OpenSSF Scorecard: Signed-Releases. Remaining weights renormalized.
Community & Adoption
Does the project have users, downloads, attention, and a welcoming setup for contributors?
12
Popularity & adoption
1Critical
How it's scored
| 0/60 | Stars — 0 stars |
| 0/25 | Forks — 0 forks |
| 0/15 | Watchers — 0 watchers |
Inputs used
| forks | 0 |
| stars | 0 |
| watchers | 0 |
| growth_state | unverified |
| growth_factor_pct | 100 |
| growth_unverified_reason | no_history |
Community health
25Critical
How it's scored
| 0/22.5 | README |
| 22.5/22.5 | License — recognized license (MIT) |
| 0/18 | CONTRIBUTING guide |
| 0/13.5 | Code of conduct |
| 0/7.2 | Issue template |
| 0/6.3 | PR template |
Inputs used
| has_readme | no |
| has_license | no |
| has_contributing | no |
| has_issue_template | no |
| has_code_of_conduct | no |
| has_pull_request_template | no |
Sustainability & Governance
Will the project survive its people — bus factor, responsiveness, who backs it, and package upkeep?
39
Maintainer resilience (bus factor)
39At risk
How it's scored
| 9/54 | Bus factor — 1 contributor(s) cover half of all commits |
| 10.4/22.5 | Commit distribution — top contributor authored 54% of commits |
| 13.5/13.5 | Contributor breadth — 10 contributors |
| 6/10 | OpenSSF Scorecard: Contributors — project has 2 contributing companies or organizations -- score normalized to 6 |
Inputs used
| bus_factor | 1 |
| contributors_sampled | 10 |
| top_contributor_share | 0.537 |
Issue & PR responsiveness
1Critical
How it's scored
| 0/46.8 | Issue resolution — no issues or no data |
| 0/38.3 | PR acceptance — no decided pull requests or no data |
| 0/15 | OpenSSF Scorecard: Code-Review — Found 0/30 approved changesets -- score normalized to 0 |
Inputs used
| merged_prs | 0 |
| open_issues | 0 |
| closed_issues | 0 |
| issue_closed_ratio | — |
| closed_unmerged_prs | 0 |
Excluded from scoring (no data or not applicable): Issue resolution, PR acceptance. Remaining weights renormalized.
Ownership & stewardship
41At risk
How it's scored
| 30/30 | Ownership backing — organization-owned |
| 0/20 | Verified domain |
| 0/25 | Owner reach — 0 followers of awee-ai |
| 11.1/25 | Track record — 10 public repos, account ~1 yr old |
Inputs used
| followers | 0 |
| owner_type | Organization |
| is_verified | — |
| owner_login | awee-ai |
| public_repos | 10 |
| account_age_days | 638 |
Package maintenance
84Good
How it's scored
| 25/25 | Published & resolvable — 1 package(s) on go |
| 35/35 | Publish recency — latest publish 7 days ago |
| 4/20 | Version history — 1 published versions |
| 20/20 | Not deprecated — active, not deprecated or yanked |
Inputs used
| packages | github.com/awee-ai/go-tokenizer |
| ecosystems | go |
| any_deprecated | no |
| min_days_since_publish | 7 |
Engineering Quality
Are baseline engineering and documentation practices in place?
22
Engineering practices
30At risk
How it's scored
| 0/24 | CI workflows |
| 24/24 | Tests present |
| 0/16 | Linter config |
| 0/9.6 | Pre-commit hooks |
| 0/6.4 | .editorconfig |
| 0/20 | OpenSSF Scorecard: CI-Tests — no data |
Inputs used
| has_ci | no |
| has_tests | yes |
| has_editorconfig | no |
| has_linter_config | no |
| has_precommit_config | no |
Excluded from scoring (no data or not applicable): OpenSSF Scorecard: CI-Tests. Remaining weights renormalized.
Documentation
10Critical
How it's scored
| 0/30 | README |
| 0/25 | Documentation directory |
| 0/15 | Documentation / homepage site |
| 0/10 | Repository description |
| 0/10 | Topics |
| 10/10 | Wiki |
Inputs used
| topics | — |
| has_wiki | yes |
| homepage | — |
| has_readme | no |
| has_docs_dir | no |
| has_description | no |
Security
Are visible security and supply-chain practices strong, without unresolved high-risk jurisdiction exposure?
28
Security posture
28Critical
How it's scored
| 7.5/7.5 | Binary-Artifacts — no binaries found in the repo |
| 0/7.5 | Branch-Protection — branch protection not enabled on development/release branches |
| 0/2.5 | CI-Tests — no data |
| 0/2.5 | CII-Best-Practices — no effort to earn an OpenSSF best practices badge detected |
| 0/7.5 | Code-Review — Found 0/30 approved changesets -- score normalized to 0 |
| 1.5/2.5 | Contributors — project has 2 contributing companies or organizations -- score normalized to 6 |
| 0/10 | Dangerous-Workflow — no data |
| 0/7.5 | Dependency-Update-Tool — no update tool detected |
| 0/5 | Fuzzing — project is not fuzzed |
| 2.5/2.5 | License — license file detected |
| 0/7.5 | Maintained — 1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 |
| 0/5 | Packaging — no data |
| 0/5 | Pinned-Dependencies — no data |
| 0/5 | SAST — no SAST tool detected |
| 0/5 | Security-Policy — security policy file not detected |
| 0/7.5 | Signed-Releases — no data |
| 0/7.5 | Token-Permissions — no data |
| 7.5/7.5 | Vulnerabilities — 0 existing vulnerabilities detected |
Inputs used
| source | openssf_scorecard |
| checks_evaluated | 12 |
| scorecard_version | v5.5.0 |
| checks_inconclusive | 6 |
| scorecard_aggregate | 2.8 |
Excluded from scoring (no data or not applicable): ci_tests, dangerous_workflow, packaging, pinned_dependencies, signed_releases, token_permissions. Remaining weights renormalized.
AI Readiness
How well is the repo equipped to be developed and maintained with AI coding agents? An independent, experimental badge — weight 0.0, so it is surfaced on its own and does not affect the overall health score.
55
Agent context & guidance
32At risk
How it's scored
| 0/45 | Agent instructions — no CLAUDE.md / AGENTS.md / editor rules |
| 0/15 | Machine-readable docs (llms.txt) |
| 31.8/40 | Legible commit history — 28 of 47 human commits state their intent (structured subject or explanatory body) |
Inputs used
| has_llms_txt | no |
| legible_history_share | 0.596 |
| agent_instruction_files | — |
| agent_instruction_max_bytes | — |
Verify loop (build / test / typecheck)
62Moderate
How it's scored
| 12.6/18 | One-command bootstrap — go.mod (toolchain convention, no task runner) |
| 22/22 | Automated tests |
| 0/11 | Lint / format config |
| 11/11 | Static type checking — Go (statically typed) |
| 10/10 | Reproducible environment — lockfile |
| 0/10 | Demonstrated agent practice — no agent-authored commits among the last 47 |
| 0/8 | Automated maintenance — no automated dependency updates observed |
| 0/10 | OpenSSF Scorecard: Pinned-Dependencies — no data |
Inputs used
| has_nix | no |
| has_tests | yes |
| lockfiles | go.sum |
| has_dockerfile | no |
| typed_language | yes |
| bootstrap_files | — |
| has_devcontainer | no |
| has_linter_config | no |
| typecheck_configs | — |
| agent_commit_share | 0 |
| toolchain_manifests | go.mod |
| dependency_bot_commit_share | 0 |
Excluded from scoring (no data or not applicable): OpenSSF Scorecard: Pinned-Dependencies. Remaining weights renormalized.
How it's scored
| 45/45 | Type-checkable code — Go (statically typed) |
| 39.3/55 | Manageable file sizes — 6/21 source files over 60KB |
Inputs used
| primary_language | Go |
| largest_source_bytes | 7,852,594 |
| source_files_sampled | 21 |
| oversized_source_files | 6 |
Key facts
0GitHub stars
10contributors
4commits, last 12 months
7days since last push
1releases
1bus factor
0open issues
Gopackage ecosystems
Data collection warnings
- Community profile unavailable
- GitHub dependency-graph SBOM unavailable (404); the dependency graph may be disabled for this repository
More detail
OpenSSF Scorecard 2.8 / 10
2.8aggregate
Independent, tool-agnostic security assessment from the open-source OpenSSF Scorecard. Each check rewards a security practice, not a specific vendor's tool. Checks Scorecard could not determine are marked n/a and excluded from the security score (never counted as zero).
| 10 | Binary-Artifacts | no binaries found in the repo |
| 0 | Branch-Protection | branch protection not enabled on development/release branches |
| n/a | CI-Tests | no pull request found |
| 0 | CII-Best-Practices | no effort to earn an OpenSSF best practices badge detected |
| 0 | Code-Review | Found 0/30 approved changesets -- score normalized to 0 |
| 6 | Contributors | project has 2 contributing companies or organizations -- score normalized to 6 |
| n/a | Dangerous-Workflow | no workflows found |
| 0 | Dependency-Update-Tool | no update tool detected |
| 0 | Fuzzing | project is not fuzzed |
| 10 | License | license file detected |
| 0 | Maintained | 1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 |
| n/a | Packaging | packaging workflow not detected |
| n/a | Pinned-Dependencies | no dependencies found |
| 0 | SAST | no SAST tool detected |
| 0 | Security-Policy | security policy file not detected |
| n/a | Signed-Releases | no releases found |
| n/a | Token-Permissions | No tokens found |
| 10 | Vulnerabilities | 0 existing vulnerabilities detected |
Direct dependencies 2
| Registry | Package | Version constraint | Manifest |
|---|---|---|---|
| Go | github.com/dlclark/regexp2 | v1.11.5 | go.mod |
| Go | github.com/stretchr/testify | v1.11.1 | go.mod |
All dependencies not collected
The resolved dependency set could not be collected for this report: GitHub dependency-graph SBOM unavailable (404); the dependency graph may be disabled for this repository
Raw JSON report machine-readable
{
"data": {
"repo": {
"topics": [],
"is_fork": true,
"size_kb": 9123,
"has_wiki": true,
"homepage": null,
"languages": {
"Go": 44449,
"Python": 437
},
"pushed_at": "2026-07-20T00:37:52Z",
"created_at": "2025-07-10T23:57:40Z",
"owner_type": "Organization",
"updated_at": "2026-07-20T00:36:54Z",
"description": "[WIP] Pure Go implementation of OpenAI's and Ollama, Anthropic tokenizers",
"is_archived": false,
"is_disabled": false,
"license_spdx": "MIT",
"default_branch": "main",
"license_spdx_raw": "MIT",
"primary_language": "Go",
"significant_languages": [
"Go"
]
},
"owner": {
"blog": "https://awee.ai",
"name": "AweeAI",
"type": "Organization",
"login": "awee-ai",
"company": null,
"location": "Lithuania",
"followers": 0,
"avatar_url": "https://avatars.githubusercontent.com/u/186479355?v=4",
"created_at": "2024-10-27T13:16:23Z",
"is_verified": null,
"public_repos": 10,
"account_age_days": 638
},
"license": {
"state": "standard",
"spdx_id": "MIT",
"raw_spdx": "MIT",
"file_present": true,
"scorecard_found": true,
"profile_has_license": false
},
"activity": {
"releases": [
{
"tag": "v0.1.0",
"kind": "minor",
"published_at": "2026-07-20T00:28:41Z"
}
],
"recent_commits": [
{
"oid": "04c4a06126a323a6105253fcbe05124f1b01756c",
"body": "Signed-off-by: Ignas Bernotas <ignas@toawe.me>",
"is_bot": false,
"headline": "chore: bump go toolchain and test dependencies",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2026-07-20T00:28:41Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "969bbb642044347150e2ae88b0d6f508b4a9e973",
"body": null,
"is_bot": false,
"headline": "feat: approx gemma/gemini model support",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-08-11T10:31:11Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "6ec759bc73960e79549c54da5862e1570f9dd466",
"body": null,
"is_bot": false,
"headline": "fix: linter issues",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-08-11T10:18:45Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "b1ec77dc381426f22990081df49ac6b268553269",
"body": null,
"is_bot": false,
"headline": "fix: switch anthropic tokenizer to cl100k_base with ratio multiplier",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-08-11T10:16:51Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e13d63d7f310e68ee9c6ba92a788f7d37290e675",
"body": null,
"is_bot": false,
"headline": "feat: anthropic",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-07-13T23:46:27Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "f9fed5a1f2670a1395a7e8bdd8b679a00fa2bd31",
"body": null,
"is_bot": false,
"headline": "feat: add anthropic tokenizer fallback",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-07-13T12:08:56Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "214cf54cfd52e9daef45db9691e63c6b3cb19d5c",
"body": null,
"is_bot": false,
"headline": "chore: rename package to github.com/awee-ai/go-tokenizer",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-07-11T17:39:23Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "b2ab4ee42fd4eae8b7575dcbc27de588ce56521e",
"body": null,
"is_bot": false,
"headline": "feat: llama tokenizer and some ollama wip fallbacks",
"author_name": "Ignas Bernotas",
"author_login": "iberflow",
"committed_at": "2025-07-11T17:35:56Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "07878231645aadb13fdea5ce1cd3603177a6ce42",
"body": "Add o3, o4-min, gpt-4.1",
"is_bot": false,
"headline": "Merge pull request #18 from ando-masaki/add-o3-and-o4-mini-fix",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2025-06-05T18:34:31Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4e6fabdf3c93d28645d98f81403f21fdea81f863",
"body": "Add o3, o4-min, gpt-4.1",
"is_bot": false,
"headline": "Update tokenizer.go",
"author_name": "ando-masaki",
"author_login": "ando-masaki",
"committed_at": "2025-06-05T06:16:33Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "d82759aa66c5ab2fbc972525d7775141b0099065",
"body": "This reverts commit a26622ed5c27d9dd14a06ac61bb5e20a7cddd2ac.",
"is_bot": false,
"headline": "Revert \"small performance improvement\"",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2025-05-09T22:24:16Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a26622ed5c27d9dd14a06ac61bb5e20a7cddd2ac",
"body": null,
"is_bot": false,
"headline": "small performance improvement",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-03-27T14:28:02Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e56ab3c0656c9d36f5a7483b5949f55ec8e280ac",
"body": null,
"is_bot": false,
"headline": "Remove defer, also small improvement using strings for bpe",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-03-27T14:18:22Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "bc8ba39dc0bbee16968778c1a6e792e79b94f486",
"body": null,
"is_bot": false,
"headline": "🚀 feat: added support for O3-mini model",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-03-07T14:50:38Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e4624aa318a9fce453eebc18536c2f96b33e35c5",
"body": null,
"is_bot": false,
"headline": "🐛 problem with regexp generated regexp code dependencies",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-02-25T15:55:43Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "8ac6fdfcb9cb83eb69eb62709090abf5a6fc6080",
"body": "feat: memory-efficient token counting functionality",
"is_bot": false,
"headline": "Merge pull request #13 from amalucelli/count",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2025-02-17T20:46:05Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "d47b68072e97bb7663b46f802b7a1835127b059a",
"body": "This uses iterators for the common logic and the function Count and Encode would do different things",
"is_bot": false,
"headline": "Refactored code to remove duplication",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-02-13T17:50:18Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "cd27b29fe9bcefbfde7b860b81776f27132b15a7",
"body": null,
"is_bot": false,
"headline": "feat: add count method for tokens",
"author_name": "Alexandre Guimarães Malucelli",
"author_login": "amalucelli",
"committed_at": "2025-02-13T17:50:16Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "03e0e5f2a364bc0069e17f15107ad16ddd443355",
"body": "pre-compile regexp2 with regexp2cg",
"is_bot": false,
"headline": "Merge pull request #11 from terwey/regexp2cg",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2025-02-12T22:59:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2e19c1421ebbcdd3d959ebd04fc8e44b70e64696",
"body": null,
"is_bot": false,
"headline": "Added a go generate command for the compiled regular expression",
"author_name": "Mariano Wahlmann",
"author_login": null,
"committed_at": "2025-02-12T22:57:06Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "025023d0144d79b96726319c4375e21a8b80837e",
"body": "Add support of chatgpt-4o-latest model",
"is_bot": false,
"headline": "Merge pull request #12 from DenisPalnitsky/chatgpt-4o-support",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2025-01-28T18:43:40Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "fdcd81a21a133e40754a66a8346439cfa3b87350",
"body": null,
"is_bot": false,
"headline": "support of chatgpt-4o-latest",
"author_name": "Denis Palnitsky",
"author_login": "DenisPalnitsky",
"committed_at": "2025-01-24T18:29:54Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "45878105be78c8394b2cbd428bbbe2049640029d",
"body": null,
"is_bot": false,
"headline": "pre-compile regexp2 with regexp2cg",
"author_name": "Yorick Terweijden",
"author_login": "terwey",
"committed_at": "2024-12-04T20:56:37Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2ad5811ad0b7d7c869c93cc33f4f161390464fe8",
"body": "Added o1-preview and o1-mini",
"is_bot": false,
"headline": "Merge pull request #10 from ando-masaki/o1-preview-and-mini-fix",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2024-11-22T22:12:55Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "60153be10d9a8246e1623eb6cf2e6e269e34bba3",
"body": null,
"is_bot": false,
"headline": "Simplified model selection",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2024-11-22T22:10:42Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "1ca7752d6d32ae47d743f933ac9c8783dd4a26c6",
"body": null,
"is_bot": false,
"headline": "Update tokenizer.go",
"author_name": "ando-masaki",
"author_login": "ando-masaki",
"committed_at": "2024-11-19T03:42:24Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "60becaa41f4a53a291b43420c9c88dac330880b5",
"body": "Update tokenizer.go",
"is_bot": false,
"headline": "Merge pull request #8 from GOOD21/main",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2024-11-07T14:32:20Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "ad6a208b2232926024e7f642fd617ff8e4ca9d71",
"body": null,
"is_bot": false,
"headline": "Update tokenizer.go",
"author_name": "GOOD21",
"author_login": "GOOD21",
"committed_at": "2024-11-07T10:09:50Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "b74b6b8b3a6705a0d7cb58ef9df45f6aba8404a4",
"body": "Fix panic caused by model name length",
"is_bot": false,
"headline": "Update tokenizer.go",
"author_name": "GOOD21",
"author_login": "GOOD21",
"committed_at": "2024-11-07T09:48:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "53141e47ff8b62f2b1f97a11f024f3cb515d09b7",
"body": "This commit should add support for all OpenAI\ncurrently available models.",
"is_bot": false,
"headline": "Added support for GPT4o and o200k_base models",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2024-09-13T15:28:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "7f65fe7d582fc7772cd74271d20dfad52fb8611a",
"body": "Generate embedded vocabulary maps lazily",
"is_bot": false,
"headline": "Merge pull request #6 from espadolini/lazy-load",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2024-05-10T19:46:47Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "93a91392ef75e411fc080030b02c448d09c8250e",
"body": null,
"is_bot": false,
"headline": "Format the code after codegen",
"author_name": "Edoardo Spadolini",
"author_login": "espadolini",
"committed_at": "2024-05-10T18:27:19Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "944cac3e81265a66bea9a725714c1698565e346c",
"body": null,
"is_bot": false,
"headline": "Generate embedded vocabulary maps lazily",
"author_name": "Edoardo Spadolini",
"author_login": "espadolini",
"committed_at": "2024-05-09T09:40:58Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "70d9b595feda6b98d6339ee4bab5f7a0c7c997f0",
"body": "Update alternatives",
"is_bot": false,
"headline": "Merge pull request #3 from philippgille/patch-1",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-19T13:07:29Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4f793dc73f725f0c821ada3bcc30a84659850a5a",
"body": null,
"is_bot": false,
"headline": "Update alternatives",
"author_name": "Philipp Gillé",
"author_login": "philippgille",
"committed_at": "2023-04-18T20:32:25Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "94e23ef2a7f6a2832126c7ace1001258220d3cd5",
"body": "Go's regexp doesn't allow for look-around so had to re-write the\nregexp but that had a bug with multiple spaces. Switched to regexp2\nthat supports the native tiktoken regexp",
"is_bot": false,
"headline": "🐛 fixed bug with spaces",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-11T13:19:07Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "876e66b44d595542c8a0251b0a5bbadfdbb9b205",
"body": "before it was not possible to install the cli",
"is_bot": false,
"headline": "🐛 moved cli into its own folder",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-11T02:28:56Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4868b0b5dc763f5886b8956dd4a9c9749aa7ae7e",
"body": "Renamed variables struct to reflect what they do a little better.",
"is_bot": false,
"headline": "🪄 clean up the bpe algorithm",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-09T21:43:58Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "12a3097aa3fad93104c64a7e774d974879e42927",
"body": "📘 fixed usage example in readme",
"is_bot": false,
"headline": "Merge pull request #1 from MarvinJWendt/patch-1",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-08T15:00:03Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2886eb0986a0e4fe3220a8430ef4ccbe1a0560b5",
"body": "- Indents in the import block were not consistent\r\n- `()` was missing in the main function\r\n- `token` was unused",
"is_bot": false,
"headline": "🛂 fixed usage example in readme",
"author_name": "Marvin Wendt",
"author_login": "MarvinJWendt",
"committed_at": "2023-04-08T02:07:20Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "859c61e7dc23560fefb62403e163a1afd80814cf",
"body": "After comfirming the expected behavior changed tests to match OpenAI's\nbehavior. It should now work as expected.",
"is_bot": false,
"headline": "🛂 fixed tests",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-07T15:06:52Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "1b538b580912c6e4e1c8e8ba72e82aece232bd9f",
"body": "this would be helpful for other people to test and use the tokenizer.\nEven from other languages because this can be distributed as a single\nbinary.",
"is_bot": false,
"headline": "🌟 added a cli to use the tokenizer via the command Lines",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T21:56:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "79aa80d63f09a0d74b360df077d204f7755bc714",
"body": "The internal generate command was using log.Fatal instead of the\ncorrect log.Fatalf.",
"is_bot": false,
"headline": "🐛 fixed an issue in the internal generate command",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T20:58:15Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "ea2b0eea6b85ee333cdf235e71442c8196c742ad",
"body": "Right now emits the same output as OpenAI's tiktoken but I'm not sure\nthat's the right output.",
"is_bot": false,
"headline": "🤫 temporarily skipping failing test until I confirm the expected output",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T20:10:08Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2e69c6a4db9547039ca536c281a73c350a20b57b",
"body": "This will make it more obvious the status of the build when reading the landing page",
"is_bot": false,
"headline": "📗 adding test results badge to README.md",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T20:02:57Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a4eb7ed16c9dbe62ab99a614157759f64fd99dac",
"body": "This should help verify future releases. The test are not passing as of now because I'm not sure tiktoken's (original python library) output is correct.",
"is_bot": false,
"headline": "🛂 add github actions",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T19:58:32Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "72848e8b530ee3bcb5f464232e39d905e06109a4",
"body": "This implementations should work for most of OpenAI's model. However\nWe are not yet dealing with the special tokens and/or gpt2 encoding.",
"is_bot": false,
"headline": "🚢 Initial Release",
"author_name": "Mariano Wahlmann",
"author_login": "bluescreen10",
"committed_at": "2023-04-06T19:53:05Z",
"body_truncated": false,
"is_coding_agent": false
}
],
"releases_count": 1,
"commits_last_year": 4,
"latest_release_at": "2026-07-20T00:28:41Z",
"latest_release_tag": "v0.1.0",
"releases_from_tags": true,
"days_since_last_push": 7,
"active_weeks_last_year": 2,
"days_since_latest_release": 7,
"mean_days_between_releases": null
},
"community": {
"has_readme": false,
"has_license": false,
"has_description": false,
"has_contributing": false,
"health_percentage": null,
"has_issue_template": false,
"has_code_of_conduct": false,
"has_pull_request_template": false
},
"ecosystem": {
"packages": [
{
"name": "github.com/awee-ai/go-tokenizer",
"exists": true,
"license": null,
"keywords": [],
"ecosystem": "go",
"matches_repo": true,
"registry_url": "https://pkg.go.dev/github.com/awee-ai/go-tokenizer",
"is_deprecated": false,
"latest_version": "v0.1.0",
"repository_url": "https://github.com/awee-ai/go-tokenizer",
"versions_count": 1,
"total_downloads": null,
"dependents_count": null,
"deprecation_note": null,
"maintainers_count": null,
"monthly_downloads": null,
"first_published_at": null,
"latest_published_at": "2026-07-20T00:28:41Z",
"latest_version_yanked": null,
"days_since_latest_publish": 7
}
]
},
"popularity": {
"forks": 0,
"stars": 0,
"watchers": 0,
"fork_history": {
"days": [],
"complete": true,
"collected": 0,
"total_forks": 0
},
"star_history": {
"days": [],
"complete": true,
"collected": 0,
"total_stars": 0,
"collected_at": null
},
"open_issues_and_prs": 0
},
"ai_readiness": {
"has_nix": false,
"example_dirs": [],
"has_llms_txt": false,
"has_dockerfile": false,
"has_mcp_signal": false,
"bootstrap_files": [],
"api_schema_files": [],
"has_devcontainer": false,
"typecheck_configs": [],
"toolchain_manifests": [
"go.mod"
],
"largest_source_bytes": 7852594,
"source_files_sampled": 21,
"oversized_source_files": 6,
"agent_instruction_files": [],
"agent_instruction_max_bytes": null
},
"dependencies": {
"manifests": [
"go.mod"
],
"advisories": {
"error": null,
"scope": null,
"source": null,
"findings": [],
"collected": false,
"malicious": [],
"truncated": false,
"by_severity": {},
"advisory_count": 0,
"affected_count": 0,
"assessed_count": 0,
"malicious_count": 0,
"assessed_package": null,
"unassessed_count": 0,
"direct_affected_count": 0
},
"ecosystems": [
"go"
],
"dependencies": [
{
"name": "github.com/dlclark/regexp2",
"manifest": "go.mod",
"ecosystem": "go",
"version_constraint": "v1.11.5"
},
{
"name": "github.com/stretchr/testify",
"manifest": "go.mod",
"ecosystem": "go",
"version_constraint": "v1.11.1"
}
],
"all_dependencies": {
"error": "GitHub dependency-graph SBOM unavailable (404); the dependency graph may be disabled for this repository",
"source": null,
"packages": [],
"collected": false,
"truncated": false,
"total_count": null,
"direct_count": null,
"indirect_count": null
}
},
"maintainership": {
"issues": {
"open_prs": 0,
"merged_prs": 0,
"open_issues": 0,
"closed_ratio": null,
"closed_issues": 0,
"closed_unmerged_prs": 0
},
"bus_factor": 1,
"bot_contributors": 0,
"top_contributors": [
{
"type": "User",
"login": "bluescreen10",
"commits": 22,
"avatar_url": "https://avatars.githubusercontent.com/u/86094?v=4"
},
{
"type": "User",
"login": "iberflow",
"commits": 8,
"avatar_url": "https://avatars.githubusercontent.com/u/402965?v=4"
},
{
"type": "User",
"login": "espadolini",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/886075?v=4"
},
{
"type": "User",
"login": "GOOD21",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/3470273?v=4"
},
{
"type": "User",
"login": "ando-masaki",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/4375029?v=4"
},
{
"type": "User",
"login": "amalucelli",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/9258376?v=4"
},
{
"type": "User",
"login": "DenisPalnitsky",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/154290?v=4"
},
{
"type": "User",
"login": "MarvinJWendt",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/31022056?v=4"
},
{
"type": "User",
"login": "philippgille",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/170670?v=4"
},
{
"type": "User",
"login": "terwey",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/1596124?v=4"
}
],
"contributors_sampled": 10,
"top_contributor_share": 0.537
},
"quality_signals": {
"has_ci": false,
"has_tests": true,
"ci_workflows": [],
"has_docs_dir": false,
"linter_configs": [],
"has_editorconfig": false,
"has_linter_config": false,
"has_precommit_config": false
},
"security_signals": {
"lockfiles": [
"go.sum"
],
"scorecard": {
"checks": [
{
"name": "Binary-Artifacts",
"score": 10,
"reason": "no binaries found in the repo",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#binary-artifacts"
},
{
"name": "Branch-Protection",
"score": 0,
"reason": "branch protection not enabled on development/release branches",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#branch-protection"
},
{
"name": "CI-Tests",
"score": null,
"reason": "no pull request found",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#ci-tests"
},
{
"name": "CII-Best-Practices",
"score": 0,
"reason": "no effort to earn an OpenSSF best practices badge detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#cii-best-practices"
},
{
"name": "Code-Review",
"score": 0,
"reason": "Found 0/30 approved changesets -- score normalized to 0",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#code-review"
},
{
"name": "Contributors",
"score": 6,
"reason": "project has 2 contributing companies or organizations -- score normalized to 6",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#contributors"
},
{
"name": "Dangerous-Workflow",
"score": null,
"reason": "no workflows found",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#dangerous-workflow"
},
{
"name": "Dependency-Update-Tool",
"score": 0,
"reason": "no update tool detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#dependency-update-tool"
},
{
"name": "Fuzzing",
"score": 0,
"reason": "project is not fuzzed",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#fuzzing"
},
{
"name": "License",
"score": 10,
"reason": "license file detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#license"
},
{
"name": "Maintained",
"score": 0,
"reason": "1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#maintained"
},
{
"name": "Packaging",
"score": null,
"reason": "packaging workflow not detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#packaging"
},
{
"name": "Pinned-Dependencies",
"score": null,
"reason": "no dependencies found",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#pinned-dependencies"
},
{
"name": "SAST",
"score": 0,
"reason": "no SAST tool detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#sast"
},
{
"name": "Security-Policy",
"score": 0,
"reason": "security policy file not detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#security-policy"
},
{
"name": "Signed-Releases",
"score": null,
"reason": "no releases found",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#signed-releases"
},
{
"name": "Token-Permissions",
"score": null,
"reason": "No tokens found",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#token-permissions"
},
{
"name": "Vulnerabilities",
"score": 10,
"reason": "0 existing vulnerabilities detected",
"documentation_url": "https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#vulnerabilities"
}
],
"commit": "04c4a06126a323a6105253fcbe05124f1b01756c",
"ran_at": "2026-07-28T00:12:34Z",
"aggregate_score": 2.8,
"scorecard_version": "v5.5.0"
},
"has_codeql_workflow": false,
"has_security_policy": false,
"has_dependabot_config": false
},
"contribution_flow": {
"collected": true,
"ci_last_run_at": null,
"oldest_open_prs": [],
"last_merged_pr_at": null,
"ci_last_conclusion": null,
"oldest_open_issues": []
}
},
"config": {
"disabled_metrics": [],
"disabled_categories": [],
"disabled_components": {}
},
"source": {
"url": "https://github.com/awee-ai/go-tokenizer",
"host": "github.com",
"name": "go-tokenizer",
"owner": "awee-ai"
},
"metrics": {
"overall": {
"key": "overall",
"band": "at_risk",
"name": "Overall health",
"note": null,
"notes": [],
"value": 32,
"inputs": {
"security": 28,
"vitality": 55,
"community": 12,
"governance": 39,
"engineering": 22
},
"components": []
},
"categories": [
{
"key": "vitality",
"band": "moderate",
"name": "Vitality",
"value": 55,
"weight": 0.22,
"metrics": [
{
"key": "development_activity",
"band": "at_risk",
"name": "Development activity",
"note": null,
"notes": [],
"value": 44,
"inputs": {
"commits_last_year": 4,
"human_commit_share": 1,
"days_since_last_push": 7,
"active_weeks_last_year": 2
},
"components": [
{
"key": "push_recency",
"name": "Push recency",
"detail": "last push 7 days ago",
"points": 36,
"status": "met",
"details": [
{
"code": "push_recency",
"params": {
"days": 7
}
}
],
"max_points": 36
},
{
"key": "commit_cadence",
"name": "Commit cadence",
"detail": "2/52 weeks with commits",
"points": 1.4,
"status": "partial",
"details": [
{
"code": "commit_cadence_weeks",
"params": {
"weeks": 2
}
}
],
"max_points": 36
},
{
"key": "commit_volume",
"name": "Commit volume",
"detail": "4 commits in the last year",
"points": 6.3,
"status": "partial",
"details": [
{
"code": "commits_last_year",
"params": {
"count": 4
}
}
],
"max_points": 18
},
{
"key": "openssf_scorecard_maintained",
"name": "OpenSSF Scorecard: Maintained",
"detail": "1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0",
"points": 0,
"status": "missed",
"details": [],
"max_points": 10
}
]
},
{
"key": "release_discipline",
"band": "good",
"name": "Release discipline",
"note": "Excluded from scoring (no data or not applicable): OpenSSF Scorecard: Signed-Releases. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"openssf_scorecard_signed_releases"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 72,
"inputs": {
"releases_count": 1,
"latest_release_tag": "v0.1.0",
"releases_from_tags": true,
"days_since_latest_release": 7,
"mean_days_between_releases": null
},
"components": [
{
"key": "ships_releases",
"name": "Ships releases",
"detail": "1 version tags (no GitHub releases)",
"points": 16.2,
"status": "partial",
"details": [
{
"code": "version_tags_no_releases",
"params": {
"count": 1
}
}
],
"max_points": 27
},
{
"key": "release_recency",
"name": "Release recency",
"detail": "latest release 7 days ago",
"points": 36,
"status": "met",
"details": [
{
"code": "release_recency",
"params": {
"days": 7
}
}
],
"max_points": 36
},
{
"key": "release_cadence",
"name": "Release cadence",
"detail": "cadence unknown (single release)",
"points": 12.6,
"status": "partial",
"details": [
{
"code": "release_cadence_unknown",
"params": {}
}
],
"max_points": 27
},
{
"key": "openssf_scorecard_signed_releases",
"name": "OpenSSF Scorecard: Signed-Releases",
"detail": "no releases found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "abandonment",
"band": "excellent",
"name": "Abandonment",
"note": null,
"notes": [],
"value": 100,
"inputs": {
"cap": null,
"state": "maintained",
"guards": [],
"signals": [],
"red_flag": false,
"multiplier_pct": 100,
"declared_reason": null,
"unverified_reason": null,
"unanswered_open_prs": null,
"unanswered_open_issues": null,
"days_since_last_merged_pr": null,
"days_since_last_human_commit": 7,
"days_since_last_human_commit_is_floor": false
},
"components": [
{
"key": "project_is_still_maintained",
"name": "Project is still maintained",
"detail": "last human commit 7 days ago",
"points": 100,
"status": "met",
"details": [
{
"code": "abandonment_maintained",
"params": {
"days": 7
}
}
],
"max_points": 100
}
]
}
],
"description": "Is the project alive — is code being written and are releases shipping?"
},
{
"key": "community",
"band": "critical",
"name": "Community & Adoption",
"value": 12,
"weight": 0.18,
"metrics": [
{
"key": "popularity",
"band": "critical",
"name": "Popularity & adoption",
"note": null,
"notes": [],
"value": 1,
"inputs": {
"forks": 0,
"stars": 0,
"watchers": 0,
"growth_state": "unverified",
"growth_factor_pct": 100,
"growth_unverified_reason": "no_history"
},
"components": [
{
"key": "stars",
"name": "Stars",
"detail": "0 stars",
"points": 0,
"status": "missed",
"details": [
{
"code": "stars",
"params": {
"count": 0
}
}
],
"max_points": 60
},
{
"key": "forks",
"name": "Forks",
"detail": "0 forks",
"points": 0,
"status": "missed",
"details": [
{
"code": "forks",
"params": {
"count": 0
}
}
],
"max_points": 25
},
{
"key": "watchers",
"name": "Watchers",
"detail": "0 watchers",
"points": 0,
"status": "missed",
"details": [
{
"code": "watchers",
"params": {
"count": 0
}
}
],
"max_points": 15
}
]
},
{
"key": "community_health",
"band": "critical",
"name": "Community health",
"note": null,
"notes": [],
"value": 25,
"inputs": {
"has_readme": false,
"has_license": false,
"has_contributing": false,
"has_issue_template": false,
"has_code_of_conduct": false,
"has_pull_request_template": false
},
"components": [
{
"key": "readme",
"name": "README",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 22.5
},
{
"key": "license",
"name": "License",
"detail": "recognized license (MIT)",
"points": 22.5,
"status": "met",
"details": [
{
"code": "license_standard",
"params": {}
},
{
"code": "license_spdx",
"params": {
"spdx": "MIT"
}
}
],
"max_points": 22.5
},
{
"key": "contributing_guide",
"name": "CONTRIBUTING guide",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 18
},
{
"key": "code_of_conduct",
"name": "Code of conduct",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 13.5
},
{
"key": "issue_template",
"name": "Issue template",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.2
},
{
"key": "pr_template",
"name": "PR template",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 6.3
}
]
}
],
"description": "Does the project have users, downloads, attention, and a welcoming setup for contributors?"
},
{
"key": "governance",
"band": "at_risk",
"name": "Sustainability & Governance",
"value": 39,
"weight": 0.24,
"metrics": [
{
"key": "maintainer_resilience",
"band": "at_risk",
"name": "Maintainer resilience (bus factor)",
"note": null,
"notes": [],
"value": 39,
"inputs": {
"bus_factor": 1,
"contributors_sampled": 10,
"top_contributor_share": 0.537
},
"components": [
{
"key": "bus_factor",
"name": "Bus factor",
"detail": "1 contributor(s) cover half of all commits",
"points": 9,
"status": "partial",
"details": [
{
"code": "bus_factor",
"params": {
"count": 1
}
}
],
"max_points": 54
},
{
"key": "commit_distribution",
"name": "Commit distribution",
"detail": "top contributor authored 54% of commits",
"points": 10.4,
"status": "partial",
"details": [
{
"code": "top_contributor_share",
"params": {
"share": 54
}
}
],
"max_points": 22.5
},
{
"key": "contributor_breadth",
"name": "Contributor breadth",
"detail": "10 contributors",
"points": 13.5,
"status": "met",
"details": [
{
"code": "contributors_sampled",
"params": {
"count": 10
}
}
],
"max_points": 13.5
},
{
"key": "openssf_scorecard_contributors",
"name": "OpenSSF Scorecard: Contributors",
"detail": "project has 2 contributing companies or organizations -- score normalized to 6",
"points": 6,
"status": "partial",
"details": [],
"max_points": 10
}
]
},
{
"key": "responsiveness",
"band": "critical",
"name": "Issue & PR responsiveness",
"note": "Excluded from scoring (no data or not applicable): Issue resolution, PR acceptance. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"issue_resolution",
"pr_acceptance"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 1,
"inputs": {
"merged_prs": 0,
"open_issues": 0,
"closed_issues": 0,
"issue_closed_ratio": null,
"closed_unmerged_prs": 0
},
"components": [
{
"key": "issue_resolution",
"name": "Issue resolution",
"detail": "no issues or no data",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_issues_or_data",
"params": {}
}
],
"max_points": 46.75
},
{
"key": "pr_acceptance",
"name": "PR acceptance",
"detail": "no decided pull requests or no data",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_decided_prs_or_data",
"params": {}
}
],
"max_points": 38.25
},
{
"key": "openssf_scorecard_code_review",
"name": "OpenSSF Scorecard: Code-Review",
"detail": "Found 0/30 approved changesets -- score normalized to 0",
"points": 0,
"status": "missed",
"details": [],
"max_points": 15
}
]
},
{
"key": "stewardship",
"band": "at_risk",
"name": "Ownership & stewardship",
"note": null,
"notes": [],
"value": 41,
"inputs": {
"followers": 0,
"owner_type": "Organization",
"is_verified": null,
"owner_login": "awee-ai",
"public_repos": 10,
"account_age_days": 638
},
"components": [
{
"key": "ownership_backing",
"name": "Ownership backing",
"detail": "organization-owned",
"points": 30,
"status": "met",
"details": [
{
"code": "owner_organization",
"params": {}
}
],
"max_points": 30
},
{
"key": "verified_domain",
"name": "Verified domain",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 20
},
{
"key": "owner_reach",
"name": "Owner reach",
"detail": "0 followers of awee-ai",
"points": 0,
"status": "missed",
"details": [
{
"code": "owner_followers",
"params": {
"count": 0,
"login": "awee-ai"
}
}
],
"max_points": 25
},
{
"key": "track_record",
"name": "Track record",
"detail": "10 public repos, account ~1 yr old",
"points": 11.1,
"status": "partial",
"details": [
{
"code": "public_repos",
"params": {
"count": 10
}
},
{
"code": "account_age_years",
"params": {
"years": 1
}
}
],
"max_points": 25
}
]
},
{
"key": "package_maintenance",
"band": "good",
"name": "Package maintenance",
"note": null,
"notes": [],
"value": 84,
"inputs": {
"packages": [
"github.com/awee-ai/go-tokenizer"
],
"ecosystems": "go",
"any_deprecated": false,
"min_days_since_publish": 7
},
"components": [
{
"key": "published_resolvable",
"name": "Published & resolvable",
"detail": "1 package(s) on go",
"points": 25,
"status": "met",
"details": [
{
"code": "packages_published",
"params": {
"count": 1,
"ecosystems": "go"
}
}
],
"max_points": 25
},
{
"key": "publish_recency",
"name": "Publish recency",
"detail": "latest publish 7 days ago",
"points": 35,
"status": "met",
"details": [
{
"code": "publish_recency",
"params": {
"days": 7
}
}
],
"max_points": 35
},
{
"key": "version_history",
"name": "Version history",
"detail": "1 published versions",
"points": 4,
"status": "partial",
"details": [
{
"code": "published_versions",
"params": {
"count": 1
}
}
],
"max_points": 20
},
{
"key": "not_deprecated",
"name": "Not deprecated",
"detail": "active, not deprecated or yanked",
"points": 20,
"status": "met",
"details": [
{
"code": "package_not_deprecated",
"params": {}
}
],
"max_points": 20
}
]
}
],
"description": "Will the project survive its people — bus factor, responsiveness, who backs it, and package upkeep?"
},
{
"key": "engineering",
"band": "critical",
"name": "Engineering Quality",
"value": 22,
"weight": 0.2,
"metrics": [
{
"key": "engineering_practices",
"band": "at_risk",
"name": "Engineering practices",
"note": "Excluded from scoring (no data or not applicable): OpenSSF Scorecard: CI-Tests. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"openssf_scorecard_ci_tests"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 30,
"inputs": {
"has_ci": false,
"has_tests": true,
"has_editorconfig": false,
"has_linter_config": false,
"has_precommit_config": false
},
"components": [
{
"key": "ci_workflows",
"name": "CI workflows",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 24
},
{
"key": "tests_present",
"name": "Tests present",
"detail": null,
"points": 24,
"status": "met",
"details": [],
"max_points": 24
},
{
"key": "linter_config",
"name": "Linter config",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 16
},
{
"key": "pre_commit_hooks",
"name": "Pre-commit hooks",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 9.6
},
{
"key": "editorconfig",
"name": ".editorconfig",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 6.4
},
{
"key": "openssf_scorecard_ci_tests",
"name": "OpenSSF Scorecard: CI-Tests",
"detail": "no pull request found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 20
}
]
},
{
"key": "documentation",
"band": "critical",
"name": "Documentation",
"note": null,
"notes": [],
"value": 10,
"inputs": {
"topics": [],
"has_wiki": true,
"homepage": null,
"has_readme": false,
"has_docs_dir": false,
"has_description": false
},
"components": [
{
"key": "readme",
"name": "README",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 30
},
{
"key": "documentation_directory",
"name": "Documentation directory",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 25
},
{
"key": "documentation_homepage_site",
"name": "Documentation / homepage site",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 15
},
{
"key": "repository_description",
"name": "Repository description",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 10
},
{
"key": "topics",
"name": "Topics",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 10
},
{
"key": "wiki",
"name": "Wiki",
"detail": null,
"points": 10,
"status": "met",
"details": [],
"max_points": 10
}
]
}
],
"description": "Are baseline engineering and documentation practices in place?"
},
{
"key": "security",
"band": "critical",
"name": "Security",
"value": 28,
"weight": 0.16,
"metrics": [
{
"key": "security_posture",
"band": "critical",
"name": "Security posture",
"note": "Excluded from scoring (no data or not applicable): CI-Tests, Dangerous-Workflow, Packaging, Pinned-Dependencies, Signed-Releases, Token-Permissions. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"ci_tests",
"dangerous_workflow",
"packaging",
"pinned_dependencies",
"signed_releases",
"token_permissions"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 28,
"inputs": {
"source": "openssf_scorecard",
"checks_evaluated": 12,
"scorecard_version": "v5.5.0",
"checks_inconclusive": 6,
"scorecard_aggregate": 2.8
},
"components": [
{
"key": "binary_artifacts",
"name": "Binary-Artifacts",
"detail": "no binaries found in the repo",
"points": 7.5,
"status": "met",
"details": [],
"max_points": 7.5
},
{
"key": "branch_protection",
"name": "Branch-Protection",
"detail": "branch protection not enabled on development/release branches",
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.5
},
{
"key": "ci_tests",
"name": "CI-Tests",
"detail": "no pull request found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 2.5
},
{
"key": "cii_best_practices",
"name": "CII-Best-Practices",
"detail": "no effort to earn an OpenSSF best practices badge detected",
"points": 0,
"status": "missed",
"details": [],
"max_points": 2.5
},
{
"key": "code_review",
"name": "Code-Review",
"detail": "Found 0/30 approved changesets -- score normalized to 0",
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.5
},
{
"key": "contributors",
"name": "Contributors",
"detail": "project has 2 contributing companies or organizations -- score normalized to 6",
"points": 1.5,
"status": "partial",
"details": [],
"max_points": 2.5
},
{
"key": "dangerous_workflow",
"name": "Dangerous-Workflow",
"detail": "no workflows found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
},
{
"key": "dependency_update_tool",
"name": "Dependency-Update-Tool",
"detail": "no update tool detected",
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.5
},
{
"key": "fuzzing",
"name": "Fuzzing",
"detail": "project is not fuzzed",
"points": 0,
"status": "missed",
"details": [],
"max_points": 5
},
{
"key": "license",
"name": "License",
"detail": "license file detected",
"points": 2.5,
"status": "met",
"details": [],
"max_points": 2.5
},
{
"key": "maintained",
"name": "Maintained",
"detail": "1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0",
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.5
},
{
"key": "packaging",
"name": "Packaging",
"detail": "packaging workflow not detected",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 5
},
{
"key": "pinned_dependencies",
"name": "Pinned-Dependencies",
"detail": "no dependencies found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 5
},
{
"key": "sast",
"name": "SAST",
"detail": "no SAST tool detected",
"points": 0,
"status": "missed",
"details": [],
"max_points": 5
},
{
"key": "security_policy",
"name": "Security-Policy",
"detail": "security policy file not detected",
"points": 0,
"status": "missed",
"details": [],
"max_points": 5
},
{
"key": "signed_releases",
"name": "Signed-Releases",
"detail": "no releases found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 7.5
},
{
"key": "token_permissions",
"name": "Token-Permissions",
"detail": "No tokens found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 7.5
},
{
"key": "vulnerabilities",
"name": "Vulnerabilities",
"detail": "0 existing vulnerabilities detected",
"points": 7.5,
"status": "met",
"details": [],
"max_points": 7.5
}
]
},
{
"key": "high_risk_jurisdiction_exposure",
"band": "excellent",
"name": "High-Risk Jurisdiction Exposure",
"note": "Only high-confidence self-published location evidence affects this multiplier. Ambiguous matches are review-only; country evidence is not proof of nationality, citizenship, legal registration, malicious intent, or sanctions status.",
"notes": [
{
"code": "jurisdiction_evidence_limits",
"params": {}
}
],
"value": 100,
"inputs": {
"meaning": "self-published location evidence; not nationality or citizenship",
"red_flag": false,
"exposures": [],
"policy_countries": [
"Russia",
"Iran",
"North Korea"
],
"review_only_matches": 0,
"assessed_self_published_locations": 17
},
"components": [
{
"key": "policy_exposure_multiplier",
"name": "Policy exposure multiplier",
"detail": "no confirmed policy-scope location match",
"points": 100,
"status": "met",
"details": [
{
"code": "jurisdiction_no_match",
"params": {}
}
],
"max_points": 100
}
]
}
],
"description": "Are visible security and supply-chain practices strong, with no malicious dependency and no unresolved high-risk jurisdiction exposure?"
},
{
"key": "ai_readiness",
"band": "moderate",
"name": "AI Readiness",
"value": 55,
"weight": 0,
"metrics": [
{
"key": "ai_agent_context",
"band": "at_risk",
"name": "Agent context & guidance",
"note": null,
"notes": [],
"value": 32,
"inputs": {
"has_llms_txt": false,
"legible_history_share": 0.596,
"agent_instruction_files": [],
"agent_instruction_max_bytes": null
},
"components": [
{
"key": "agent_instructions",
"name": "Agent instructions",
"detail": "no CLAUDE.md / AGENTS.md / editor rules",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_agent_instructions",
"params": {}
}
],
"max_points": 45
},
{
"key": "machine_readable_docs_llms_txt",
"name": "Machine-readable docs (llms.txt)",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 15
},
{
"key": "legible_commit_history",
"name": "Legible commit history",
"detail": "28 of 47 human commits state their intent (structured subject or explanatory body)",
"points": 31.8,
"status": "partial",
"details": [
{
"code": "legible_history",
"params": {
"legible": 28,
"sampled": 47
}
}
],
"max_points": 40
}
]
},
{
"key": "ai_verify_loop",
"band": "moderate",
"name": "Verify loop (build / test / typecheck)",
"note": "Excluded from scoring (no data or not applicable): OpenSSF Scorecard: Pinned-Dependencies. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"openssf_scorecard_pinned_dependencies"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 62,
"inputs": {
"has_nix": false,
"has_tests": true,
"lockfiles": [
"go.sum"
],
"has_dockerfile": false,
"typed_language": true,
"bootstrap_files": [],
"has_devcontainer": false,
"has_linter_config": false,
"typecheck_configs": [],
"agent_commit_share": 0,
"toolchain_manifests": [
"go.mod"
],
"dependency_bot_commit_share": 0
},
"components": [
{
"key": "one_command_bootstrap",
"name": "One-command bootstrap",
"detail": "go.mod (toolchain convention, no task runner)",
"points": 12.6,
"status": "partial",
"details": [
{
"code": "toolchain_convention",
"params": {
"files": "go.mod"
}
}
],
"max_points": 18
},
{
"key": "automated_tests",
"name": "Automated tests",
"detail": null,
"points": 22,
"status": "met",
"details": [],
"max_points": 22
},
{
"key": "lint_format_config",
"name": "Lint / format config",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 11
},
{
"key": "static_type_checking",
"name": "Static type checking",
"detail": "Go (statically typed)",
"points": 11,
"status": "met",
"details": [
{
"code": "statically_typed_language",
"params": {
"language": "Go"
}
}
],
"max_points": 11
},
{
"key": "reproducible_environment",
"name": "Reproducible environment",
"detail": "lockfile",
"points": 10,
"status": "met",
"details": [
{
"code": "file_list",
"params": {
"files": "lockfile"
}
}
],
"max_points": 10
},
{
"key": "demonstrated_agent_practice",
"name": "Demonstrated agent practice",
"detail": "no agent-authored commits among the last 47",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_agent_authored_commits",
"params": {
"sampled": 47
}
}
],
"max_points": 10
},
{
"key": "automated_maintenance",
"name": "Automated maintenance",
"detail": "no automated dependency updates observed",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_dependency_automation",
"params": {}
}
],
"max_points": 8
},
{
"key": "openssf_scorecard_pinned_dependencies",
"name": "OpenSSF Scorecard: Pinned-Dependencies",
"detail": "no dependencies found",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "ai_code_legibility",
"band": "good",
"name": "Code legibility for models",
"note": null,
"notes": [],
"value": 84,
"inputs": {
"primary_language": "Go",
"largest_source_bytes": 7852594,
"source_files_sampled": 21,
"oversized_source_files": 6
},
"components": [
{
"key": "type_checkable_code",
"name": "Type-checkable code",
"detail": "Go (statically typed)",
"points": 45,
"status": "met",
"details": [
{
"code": "statically_typed_language",
"params": {
"language": "Go"
}
}
],
"max_points": 45
},
{
"key": "manageable_file_sizes",
"name": "Manageable file sizes",
"detail": "6/21 source files over 60KB",
"points": 39.3,
"status": "partial",
"details": [
{
"code": "oversized_source_files",
"params": {
"kb": 60,
"sampled": 21,
"oversized": 6
}
}
],
"max_points": 55
}
]
}
],
"description": "How well is the repo equipped to be developed and maintained with AI coding agents? An independent, experimental badge — weight 0.0, so it is surfaced on its own and does not affect the overall health score."
}
],
"metrics_version": "1.13.0"
},
"warnings": [
"Community profile unavailable",
"GitHub dependency-graph SBOM unavailable (404); the dependency graph may be disabled for this repository"
],
"report_type": "repository",
"generated_at": "2026-07-28T00:12:43.339053Z",
"schema_version": "0.27.0",
"badge_url": "https://raw.githubusercontent.com/inspect-software/badges/main/v1/a/awee-ai/go-tokenizer.svg",
"full_name": "awee-ai/go-tokenizer",
"license_state": "standard",
"license_spdx": "MIT"
}Scores are signals, not warranties. They reflect publicly visible practices on GitHub — not a code audit, and not a security guarantee.
Missing data is excluded and weights renormalized, never scored as zero. Methodology is versioned and open: metrics v1.13.0, schema v0.27.0 — full methodology · metrics wiki.
How one result sits in the wider record: aggregate statistics — Go.