Public recordWeak
Software health reportschema 0.27.0 · metrics 2.3.1 · 2026-07-29 01:32 UTC
METR / task-standard
METR Task Standard
METR/task-standard holds a health index of 36 out of 100, placing it in the Weak band. It scores highest on Community & Adoption (65/100) and lowest on Vitality (28/100). It was last updated 540 days ago. 2 contributors account for most of its recent work.
36
overall / 100Software health index
Metrics are grouped into weighted categories on one standardized 1–100 scale. Overall starts as their weighted mean, calibrated against the distribution of the public record so bands carry percentile meaning; when public evidence triggers the High-Risk Jurisdiction Policy, the rating is adjusted and receives an At Risk ceiling of 34.
36
Exceptional93-100The record's top tier (≈ top 5%); essentially all checked criteria met
Excellent80-92Strong across the board; minor gaps
Good65-79Healthy; gaps are limited and manageable
Moderate50-64Acceptable with notable gaps; review recommended
Weak35-49Material weaknesses across several areas
At Risk20-34Significant weaknesses; adoption warrants caution
Critical1-19Severe problems (abandoned, single-maintainer, no hygiene)
Score profile
Each axis is a category. The shape matters more than the average — a healthy subject fills the whole shape, while a spike-and-crater profile means strength in one dimension is masking risk in another.
The weighted overall 45 is calibrated to 42 on the published index scale (record calibration 2026-08-02).
Ownership
METROrganization
452 followers62 public repossince Oct 2022
This repository is backed by an organization — shared, accountable stewardship that can outlive any single maintainer.
Metrics by category
Vitality
Is the project alive — is code being written and are releases shipping?
28
Development activity
1Critical
Abandonment alert
Work has arrived and nobody has attended to it
no human commit in 545 daysautomated commits excludeda published fix left unappliedreleases stalled
Silence alone is never the finding — a small, finished library may need no commits for years. This alert requires both: no human commit for a long time, and obligations the project is visibly not discharging. Automated commits do not count as maintenance.
How abandonment is assessedHow it's scored
| 0/36 | Push recency — last push 540 days ago |
| 0/36 | Commit cadence — 0/52 weeks with commits |
| 0/18 | Commit volume — 0 commits in the last year |
| 0/10 | OpenSSF Scorecard: Maintained — no data |
Inputs used
| commits_last_year | 0 |
| human_commit_share | 1 |
| days_since_last_push | 540 |
| active_weeks_last_year | 0 |
Release discipline
68Good
How it's scored
| 27/27 | Ships releases — 6 releases published |
| 7.2/36 | Release recency — latest release 728 days ago |
| 27/27 | Release cadence — a release every ~30.4 days |
| 0/10 | OpenSSF Scorecard: Signed-Releases — no data |
Inputs used
| releases_count | 6 |
| latest_release_tag | v0.3.0 |
| releases_from_tags | no |
| days_since_latest_release | 728 |
| mean_days_between_releases | 30.4 |
Community & Adoption
Does the project have users, downloads, attention, and a welcoming setup for contributors?
65
Popularity & adoption
54Moderate
How it's scored
| 36.8/60 | Stars — 186 stars |
| 13/25 | Forks — 37 forks |
| 4.3/15 | Watchers — 7 watchers |
Inputs used
| forks | 37 |
| stars | 186 |
| watchers | 7 |
| growth_state | unverified |
| growth_factor_pct | 100 |
| growth_unverified_reason | no_history |
Community health
77Good
How it's scored
| 22.5/22.5 | README |
| 22.5/22.5 | License — recognized license (MIT) |
| 18/18 | CONTRIBUTING guide |
| 0/13.5 | Code of conduct |
| 0/7.2 | Issue template |
| 6.3/6.3 | PR template |
Inputs used
| has_readme | yes |
| has_license | yes |
| readme_badges | — |
| has_contributing | yes |
| has_issue_template | no |
| has_code_of_conduct | no |
| readme_badge_services | — |
| has_pull_request_template | yes |
Sustainability & Governance
Will the project survive its people — bus factor, responsiveness, who backs it, and package upkeep?
63
Maintainer resilience (bus factor)
61Moderate
How it's scored
| 25.2/54 | Bus factor — 2 contributor(s) cover half of all commits |
| 16.2/22.5 | Commit distribution — top contributor authored 28% of commits |
| 13.5/13.5 | Contributor breadth — 12 contributors |
| 0/10 | OpenSSF Scorecard: Contributors — no data |
Inputs used
| bus_factor | 2 |
| contributors_sampled | 12 |
| top_contributor_share | 0.278 |
Issue & PR responsiveness
58Moderate
How it's scored
| 16.2/42 | Issue resolution — 38% of issues closed |
| 25.2/30 | PR acceptance — 21/25 decided PRs merged |
| 0/13 | Newcomer PR acceptance — no first-time contributor's PR decided in 30d |
| 0/15 | OpenSSF Scorecard: Code-Review — no data |
Inputs used
| merged_prs | 21 |
| open_issues | 8 |
| closed_issues | 5 |
| prs_merged_7d | — |
| prs_decided_7d | — |
| prs_merged_30d | — |
| prs_decided_30d | — |
| issue_closed_ratio | 0.385 |
| closed_unmerged_prs | 4 |
| first_time_authors_30d | — |
| first_time_prs_merged_30d | — |
| first_time_prs_decided_30d | — |
Excluded from scoring (no data or not applicable): newcomer_pr_acceptance. Remaining weights renormalized.
Ownership & stewardship
70Good
How it's scored
| 30/30 | Ownership backing — organization-owned |
| 0/20 | Verified domain |
| 19.1/25 | Owner reach — 452 followers of METR |
| 20.6/25 | Track record — 62 public repos, account ~3 yr old |
Inputs used
| followers | 452 |
| owner_type | Organization |
| is_verified | — |
| owner_login | METR |
| public_repos | 62 |
| account_age_days | 1,387 |
Engineering Quality
Are baseline engineering and documentation practices in place?
34
Engineering practices
30At Risk
How it's scored
| 0/24 | CI workflows |
| 24/24 | Tests present |
| 0/16 | Linter config |
| 0/9.6 | Pre-commit hooks |
| 0/6.4 | .editorconfig |
| 0/20 | OpenSSF Scorecard: CI-Tests — no data |
Inputs used
| has_ci | no |
| has_tests | yes |
| has_editorconfig | no |
| has_linter_config | no |
| has_precommit_config | no |
Documentation
40Weak
How it's scored
| 30/30 | README |
| 0/25 | Documentation directory |
| 0/15 | Documentation / homepage site |
| 10/10 | Repository description |
| 0/10 | Topics |
| 0/10 | Wiki |
Inputs used
| topics | — |
| has_wiki | no |
| homepage | — |
| has_readme | yes |
| has_docs_dir | no |
| has_description | yes |
Security
Are visible security and supply-chain practices strong, without unresolved high-risk jurisdiction exposure?
33
Security posture
25At Risk
How it's scored
| 0/30 | Security policy (SECURITY.md) |
| 0/25 | Dependabot config |
| 25/25 | Dependency lockfiles — package-lock.json |
| 0/20 | CodeQL workflow |
Inputs used
| source | file_signals |
| lockfiles | package-lock.json |
| manifests | drivers/package.json, python-package/setup.py, workbench/package.json |
| has_codeql_workflow | no |
| has_security_policy | no |
| has_dependabot_config | no |
Dependency advisories
66Good
How it's scored
| 17.8/35 | Direct dependencies free of known advisories — 2 affected: esbuild 0.21.4 (moderate 5.3), zod 3.21.4 (moderate 5.3) |
| 0/25 | Indirect dependencies free of known advisories — transitive set not separable from development and test dependencies in this scope |
| 31.6/40 | No advisories left outstanding — 2 advisory-carrying package(s) unaddressed past 90 days; oldest published 1,034 days ago |
Inputs used
| source | osv |
| advisories | 41 |
| affected_packages | 8 |
| assessed_packages | 137 |
| unassessed_packages | 22 |
| affected_by_severity | critical 2, high 2, moderate 3, low 1 |
| direct_affected_packages | 2 |
Excluded from scoring (no data or not applicable): Indirect dependencies free of known advisories. Remaining weights renormalized. Matched 137 resolved dependencies against OSV. 22 could not be assessed — no resolved version, an unsupported ecosystem, or beyond the reported package list. This repository publishes no package the index resolves, so the repository dependency graph was assessed instead. That graph mixes development and test pins with shipped dependencies, so only the declared runtime dependencies are scored; transitive findings are reported as context and excluded from the score. Reachability is not analyzed.
AI Readiness
How well is the repo equipped to be developed and maintained with AI coding agents? Carries a deliberately small weight (4%): agent tooling is a real maintenance signal, but a repository with none can still reach 100/100.
47
Agent context & guidance
23At Risk
How it's scored
| 0/45 | Agent instructions — no CLAUDE.md / AGENTS.md / editor rules |
| 0/15 | Machine-readable docs (llms.txt) |
| 22.6/40 | Legible commit history — 31 of 73 human commits state their intent (structured subject or explanatory body) |
Inputs used
| has_llms_txt | no |
| legible_history_share | 0.425 |
| agent_instruction_files | — |
| agent_instruction_max_bytes | — |
How it's scored
| 0/18 | One-command bootstrap |
| 22/22 | Automated tests |
| 0/11 | Lint / format config |
| 11/11 | Static type checking — drivers/tsconfig.json, workbench/tsconfig.json |
| 10/10 | Reproducible environment — Dockerfile, lockfile |
| 0/10 | Demonstrated agent practice — no agent-authored commits among the last 73 |
| 0/8 | Automated maintenance — no automated dependency updates observed |
| 0/10 | OpenSSF Scorecard: Pinned-Dependencies — no data |
Inputs used
| has_nix | no |
| has_tests | yes |
| lockfiles | package-lock.json |
| has_dockerfile | yes |
| typed_language | yes |
| bootstrap_files | — |
| has_devcontainer | no |
| has_linter_config | no |
| typecheck_configs | drivers/tsconfig.json, workbench/tsconfig.json |
| agent_commit_share | 0 |
| toolchain_manifests | — |
| dependency_bot_commit_share | 0 |
Code legibility for models
100Exceptional
How it's scored
| 45/45 | Type-checkable code — TypeScript (statically typed) |
| 55/55 | Manageable file sizes — 0/76 source files over 60KB |
Inputs used
| primary_language | TypeScript |
| largest_source_bytes | 29,920 |
| source_files_sampled | 76 |
| oversized_source_files | 0 |
How it's scored
| 0/40 | API schema (OpenAPI/GraphQL/proto) |
| 0/20 | MCP server |
| 40/40 | Runnable examples — examples |
Inputs used
| example_dirs | examples |
| has_mcp_signal | no |
| api_schema_files | — |
Key facts
186GitHub stars
12contributors
0commits, last 12 months
540days since last push
6releases
2bus factor
8open issues
npm, PyPIpackage ecosystems
Data collection warnings
- Star history unavailable: GitHub GraphQL error: Resource not accessible by personal access token
- Could not fetch npm package 'task-standard-drivers' from its registry
- Could not fetch npm package 'task-standard-workbench' from its registry
- Could not fetch pypi package 'metr-task-standard' from its registry
- OpenSSF Scorecard did not return a usable result (2026/07/29 01:31:49 Warning: PATs stored in env variables GITHUB_AUTH_TOKEN and GITHUB_TOKEN differ. Scorecard will use the former.); skipping Scorecard checks
More detail
Star and fork history 0 ★ / 37 ⇿
0Stars
37Forks
6Releases
When each star and fork was added, collected from GitHub and bucketed by day. Cumulative growth sits directly above the daily additions it is made of, so the two read against each other: steady organic accretion looks nothing like an abrupt, short-lived burst. Where that difference is measurable, it is reported as growth authenticity.
Major 0Minor 2Patch 4
Each point covers 3 days.
Direct dependencies 9
| Registry | Package | Version constraint | Manifest |
|---|---|---|---|
| npm | @aws-sdk/client-ec2 | ^3.515.0 | drivers/package.json |
| npm | @types/node | ^20.14.7 | drivers/package.json |
| npm | json5 | ^2.2.3 | drivers/package.json |
| npm | object-hash | ^3.0.0 | drivers/package.json |
| npm | zod | ^3.21.4 | drivers/package.json |
| npm | esbuild | ^0.21.0 | workbench/package.json |
| npm | esbuild-runner | ^2.2.2 | workbench/package.json |
| npm | zod-to-json-schema | ^3.23.3 | workbench/package.json |
| npm | yargs | ^17.7.2 | workbench/package.json |
All dependencies 159
Full resolved dependency set from the GitHub dependency graph: 8 direct and 151 indirect (transitive) packages. The transitive closure is complete when the repository commits a lockfile.
| Registry | Package | Version | Relation |
|---|---|---|---|
| npm | @aws-sdk/client-ec2 | 3.515.0 | direct |
| npm | @types/node | 20.12.12 | direct |
| npm | @types/node | 20.14.7 | direct |
| npm | esbuild | 0.21.4 | direct |
| npm | esbuild-runner | 2.2.2 | direct |
| npm | object-hash | 3.0.0 | direct |
| npm | yargs | 17.7.2 | direct |
| npm | zod | 3.21.4 | direct |
| npm | @aws-crypto/crc32 | 3.0.0 | indirect |
| npm | @aws-crypto/ie11-detection | 3.0.0 | indirect |
| npm | @aws-crypto/sha256-browser | 3.0.0 | indirect |
| npm | @aws-crypto/sha256-js | 3.0.0 | indirect |
| npm | @aws-crypto/supports-web-crypto | 3.0.0 | indirect |
| npm | @aws-crypto/util | 3.0.0 | indirect |
| npm | @aws-sdk/client-sso | 3.515.0 | indirect |
| npm | @aws-sdk/client-sso-oidc | 3.515.0 | indirect |
| npm | @aws-sdk/client-sts | 3.515.0 | indirect |
| npm | @aws-sdk/core | 3.513.0 | indirect |
| npm | @aws-sdk/credential-provider-env | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-http | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-ini | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-node | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-process | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-sso | 3.515.0 | indirect |
| npm | @aws-sdk/credential-provider-web-identity | 3.515.0 | indirect |
| npm | @aws-sdk/middleware-host-header | 3.515.0 | indirect |
| npm | @aws-sdk/middleware-logger | 3.515.0 | indirect |
| npm | @aws-sdk/middleware-recursion-detection | 3.515.0 | indirect |
| npm | @aws-sdk/middleware-sdk-ec2 | 3.515.0 | indirect |
| npm | @aws-sdk/middleware-user-agent | 3.515.0 | indirect |
| npm | @aws-sdk/region-config-resolver | 3.515.0 | indirect |
| npm | @aws-sdk/token-providers | 3.515.0 | indirect |
| npm | @aws-sdk/types | 3.515.0 | indirect |
| npm | @aws-sdk/util-endpoints | 3.515.0 | indirect |
| npm | @aws-sdk/util-format-url | 3.515.0 | indirect |
| npm | @aws-sdk/util-locate-window | 3.495.0 | indirect |
| npm | @aws-sdk/util-user-agent-browser | 3.515.0 | indirect |
| npm | @aws-sdk/util-user-agent-node | 3.515.0 | indirect |
| npm | @aws-sdk/util-utf8-browser | 3.259.0 | indirect |
| npm | @esbuild/aix-ppc64 | 0.21.4 | indirect |
| npm | @esbuild/android-arm | 0.21.4 | indirect |
| npm | @esbuild/android-arm64 | 0.21.4 | indirect |
| npm | @esbuild/android-x64 | 0.21.4 | indirect |
| npm | @esbuild/darwin-arm64 | 0.21.4 | indirect |
| npm | @esbuild/darwin-x64 | 0.21.4 | indirect |
| npm | @esbuild/freebsd-arm64 | 0.21.4 | indirect |
| npm | @esbuild/freebsd-x64 | 0.21.4 | indirect |
| npm | @esbuild/linux-arm | 0.21.4 | indirect |
| npm | @esbuild/linux-arm64 | 0.21.4 | indirect |
| npm | @esbuild/linux-ia32 | 0.21.4 | indirect |
| npm | @esbuild/linux-loong64 | 0.21.4 | indirect |
| npm | @esbuild/linux-mips64el | 0.21.4 | indirect |
| npm | @esbuild/linux-ppc64 | 0.21.4 | indirect |
| npm | @esbuild/linux-riscv64 | 0.21.4 | indirect |
| npm | @esbuild/linux-s390x | 0.21.4 | indirect |
| npm | @esbuild/linux-x64 | 0.21.4 | indirect |
| npm | @esbuild/netbsd-x64 | 0.21.4 | indirect |
| npm | @esbuild/openbsd-x64 | 0.21.4 | indirect |
| npm | @esbuild/sunos-x64 | 0.21.4 | indirect |
| npm | @esbuild/win32-arm64 | 0.21.4 | indirect |
| npm | @esbuild/win32-ia32 | 0.21.4 | indirect |
| npm | @esbuild/win32-x64 | 0.21.4 | indirect |
| npm | @smithy/abort-controller | 2.1.3 | indirect |
| npm | @smithy/config-resolver | 2.1.4 | indirect |
| npm | @smithy/core | 1.3.5 | indirect |
| npm | @smithy/credential-provider-imds | 2.2.4 | indirect |
| npm | @smithy/eventstream-codec | 2.1.3 | indirect |
| npm | @smithy/fetch-http-handler | 2.4.3 | indirect |
| npm | @smithy/hash-node | 2.1.3 | indirect |
| npm | @smithy/invalid-dependency | 2.1.3 | indirect |
| npm | @smithy/is-array-buffer | 2.1.1 | indirect |
| npm | @smithy/middleware-content-length | 2.1.3 | indirect |
| npm | @smithy/middleware-endpoint | 2.4.4 | indirect |
| npm | @smithy/middleware-retry | 2.1.4 | indirect |
| npm | @smithy/middleware-serde | 2.1.3 | indirect |
| npm | @smithy/middleware-stack | 2.1.3 | indirect |
| npm | @smithy/node-config-provider | 2.2.4 | indirect |
| npm | @smithy/node-http-handler | 2.4.1 | indirect |
| npm | @smithy/property-provider | 2.1.3 | indirect |
| npm | @smithy/protocol-http | 3.2.1 | indirect |
| npm | @smithy/querystring-builder | 2.1.3 | indirect |
| npm | @smithy/querystring-parser | 2.1.3 | indirect |
| npm | @smithy/service-error-classification | 2.1.3 | indirect |
| npm | @smithy/shared-ini-file-loader | 2.3.4 | indirect |
| npm | @smithy/signature-v4 | 2.1.3 | indirect |
| npm | @smithy/smithy-client | 2.4.2 | indirect |
| npm | @smithy/types | 2.10.1 | indirect |
| npm | @smithy/url-parser | 2.1.3 | indirect |
| npm | @smithy/util-base64 | 2.1.1 | indirect |
| npm | @smithy/util-body-length-browser | 2.1.1 | indirect |
| npm | @smithy/util-body-length-node | 2.2.1 | indirect |
| npm | @smithy/util-buffer-from | 2.1.1 | indirect |
| npm | @smithy/util-config-provider | 2.2.1 | indirect |
| npm | @smithy/util-defaults-mode-browser | 2.1.4 | indirect |
| npm | @smithy/util-defaults-mode-node | 2.2.3 | indirect |
| npm | @smithy/util-endpoints | 1.1.4 | indirect |
| npm | @smithy/util-hex-encoding | 2.1.1 | indirect |
| npm | @smithy/util-middleware | 2.1.3 | indirect |
| npm | @smithy/util-retry | 2.1.3 | indirect |
| npm | @smithy/util-stream | 2.1.3 | indirect |
| npm | @smithy/util-uri-escape | 2.1.1 | indirect |
| npm | @smithy/util-utf8 | 2.1.1 | indirect |
| npm | @smithy/util-waiter | 2.1.3 | indirect |
| npm | @types/object-hash | 3.0.6 | indirect |
| npm | @types/yargs | 17.0.32 | indirect |
| npm | @types/yargs-parser | 21.0.3 | indirect |
| npm | ansi-regex | 5.0.1 | indirect |
| npm | ansi-styles | 4.3.0 | indirect |
| npm | bowser | 2.11.0 | indirect |
| npm | buffer-from | 1.1.2 | indirect |
| npm | cliui | 8.0.1 | indirect |
| npm | color-convert | 2.0.1 | indirect |
| npm | color-name | 1.1.4 | indirect |
| npm | emoji-regex | 8.0.0 | indirect |
| npm | escalade | 3.1.2 | indirect |
| npm | fast-xml-parser | 4.2.5 | indirect |
| npm | get-caller-file | 2.0.5 | indirect |
| npm | is-fullwidth-code-point | 3.0.0 | indirect |
| npm | prettier-no-jsx-parens | 2.8.4 | indirect |
| npm | require-directory | 2.1.1 | indirect |
| npm | source-map | 0.6.1 | indirect |
| npm | source-map-support | 0.5.21 | indirect |
| npm | string-width | 4.2.3 | indirect |
| npm | strip-ansi | 6.0.1 | indirect |
| npm | strnum | 1.0.5 | indirect |
| npm | tslib | 1.14.1 | indirect |
| npm | tslib | 2.4.0 | indirect |
| npm | tslib | 2.6.2 | indirect |
| npm | undici-types | 5.26.5 | indirect |
| npm | uuid | 8.3.2 | indirect |
| npm | uuid | 9.0.1 | indirect |
| npm | wrap-ansi | 7.0.0 | indirect |
| npm | y18n | 5.0.8 | indirect |
| npm | yargs-parser | 21.1.1 | indirect |
| PyPI | anthropic | — | indirect |
| PyPI | beautifulsoup4 | — | indirect |
| PyPI | bokeh | — | indirect |
| PyPI | cachetools | — | indirect |
| PyPI | datasets | 2.15.0 | indirect |
| PyPI | ffmpeg-python | — | indirect |
| PyPI | ipython | — | indirect |
| PyPI | langchain | — | indirect |
| PyPI | matplotlib | — | indirect |
| PyPI | numpy | — | indirect |
| PyPI | openai | — | indirect |
| PyPI | pandas | — | indirect |
| PyPI | plotly | — | indirect |
| PyPI | pydantic | — | indirect |
| PyPI | pypdf2 | 3.0.1 | indirect |
| PyPI | pytest | — | indirect |
| PyPI | python-dotenv | — | indirect |
| PyPI | scikit-learn | — | indirect |
| PyPI | sortedcontainers | — | indirect |
| PyPI | speechrecognition | — | indirect |
| PyPI | tabulate | — | indirect |
| PyPI | textract | — | indirect |
| PyPI | tiktoken | — | indirect |
| PyPI | torch | 2.1.2+cpu.cxx11.abi | indirect |
| PyPI | xgboost | — | indirect |
Dependency advisories 8
This repository publishes no package the index resolves, so its own dependency graph was assessed — 137 packages, which also include development and test pins that never ship: 8 carry known advisories, of which 2 are direct. 22 could not be assessed — no resolved version, an unsupported ecosystem, or beyond the reported package list.
| Package | Version | Relation | Severity | Advisories | Fixed in |
|---|---|---|---|---|---|
| fast-xml-parser | 4.2.5 | indirect | critical | 6 | 5.7.0 |
| torch | 2.1.2+cpu.cxx11.abi | indirect | critical | 28 | 2.13.0 |
| uuid | 8.3.2 | indirect | high | 1 | 13.0.1 |
| uuid | 9.0.1 | indirect | high | 1 | 13.0.1 |
| esbuild | 0.21.4 | direct | moderate | 1 | 0.25.0 |
| zod | 3.21.4 | direct | moderate | 1 | 3.22.3 |
| pypdf2 | 3.0.1 | indirect | moderate | 2 | 3.9.0 |
| @smithy/config-resolver | 2.1.4 | indirect | low | 1 | 4.4.0 |
An advisory means the version recorded in the dependency graph falls inside an advisory’s affected range. Reachability is not analysed, and the graph includes development and test pins — a finding may concern tooling rather than shipped software.
Raw JSON report machine-readable
{
"data": {
"repo": {
"topics": [],
"is_fork": false,
"size_kb": 13639,
"has_wiki": false,
"homepage": null,
"languages": {
"Shell": 134,
"Python": 68563,
"Dockerfile": 8712,
"TypeScript": 95233
},
"pushed_at": "2025-02-03T21:50:42Z",
"created_at": "2024-01-31T19:15:24Z",
"owner_type": "Organization",
"updated_at": "2026-07-26T18:23:49Z",
"description": "METR Task Standard",
"is_archived": false,
"is_disabled": false,
"license_spdx": "MIT",
"default_branch": "main",
"license_spdx_raw": "MIT",
"primary_language": "TypeScript",
"significant_languages": [
"TypeScript",
"Python"
]
},
"owner": {
"blog": "https://metr.org",
"name": "METR",
"type": "Organization",
"login": "METR",
"company": null,
"location": null,
"followers": 452,
"avatar_url": "https://avatars.githubusercontent.com/u/115503932?v=4",
"created_at": "2022-10-10T20:10:42Z",
"is_verified": null,
"public_repos": 62,
"account_age_days": 1387
},
"license": {
"state": "standard",
"spdx_id": "MIT",
"raw_spdx": "MIT",
"file_present": true,
"scorecard_found": null,
"profile_has_license": true
},
"activity": {
"releases": [
{
"tag": "v0.3.0",
"kind": "minor",
"published_at": "2024-07-31T00:57:14Z"
},
{
"tag": "v0.2.3",
"kind": "patch",
"published_at": "2024-05-13T22:44:03Z"
},
{
"tag": "v0.2.2",
"kind": "patch",
"published_at": "2024-04-23T22:55:40Z"
},
{
"tag": "v0.2.1",
"kind": "patch",
"published_at": "2024-03-11T21:11:36Z"
},
{
"tag": "v0.2.0",
"kind": "minor",
"published_at": "2024-03-06T21:31:35Z"
},
{
"tag": "v0.1.1",
"kind": "patch",
"published_at": "2024-02-29T20:28:48Z"
}
],
"recent_commits": [
{
"oid": "03236e9a1a0d3c9f9d63f6c9e60a9278a59d22ff",
"body": null,
"is_bot": false,
"headline": "Update CONTRIBUTING.md",
"author_name": "Thomas Broadley",
"author_login": "tbroadley",
"committed_at": "2025-02-03T21:50:42Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0715f259710a3ce835200d6aea753c8c4f691f0e",
"body": null,
"is_bot": false,
"headline": "Update license year",
"author_name": "Thomas Broadley",
"author_login": "tbroadley",
"committed_at": "2025-01-03T17:47:24Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0beaf93c67d65032a2ef0c475edce6568f4ee2fe",
"body": "Project import generated by Copybara.\r\n\r\n---------\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>\r\nCo-authored-by: Maksym Taran <maksym@metr.org>",
"is_bot": false,
"headline": "Sync task-standard 0.5.0 (#36)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-10-30T22:47:29Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e9ff50e2dfbacbe383a25e422a1e6eeb774e731b",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard v0.4.1 (#34)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-09-27T23:47:39Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "11433a0a970c062dc7533913335205337093dd89",
"body": "…(#27)\n\nIn a private task, we need to specify a fractional amount of RAM - but\r\nthis is apparently not allowed by the task standard. I think it would\r\nmake more sense to make the ram gib value a float if possible.",
"is_bot": false,
"headline": "Change ram gib range from `tuple[int, int]` to `tuple[float, float]` …",
"author_name": "sdtblckgov",
"author_login": "sdtblckgov",
"committed_at": "2024-09-20T21:50:37Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "c1871ba7e04e11b05ef2fbd9568252642af51a71",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard v0.4.0 (#32)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-08-28T01:42:42Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "d88bd7695e931a39cf2df487cbbdb803b89ba0fa",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#28)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-08-20T00:03:03Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0d91dd4662ba52214082a2ca5e69163df0c85622",
"body": "Fixes #25 for Python 3.11.\r\n\r\nI have not tested Python 3.10.",
"is_bot": false,
"headline": "Python 3.11 support (#29)",
"author_name": "art-dsit",
"author_login": "art-dsit",
"committed_at": "2024-08-13T16:38:06Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "67db1f5a2f34edae7c2ade023b84dbb5d5bb9f81",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#26)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-07-31T00:56:49Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "57cdf86a63b2f13d91ab149260cbfbc8e7d5f074",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#24)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-07-18T23:10:00Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e037d898f2fa45fe66dec39ff248da563dfe77a9",
"body": "Project import generated by Copybara.\r\n\r\n- When building an aux VM image, use the same instance to build the AMI\r\nas to run it, instead of using an instance of a fixed size with the same\r\nCPU architecture. In general, this should speed up aux VM image builds\r\n- Update the workbench's documentation\r\n- Fix a bug in the `compare-llms` example agent that was preventing it\r\nfrom running\r\n- Allow running `npm run agent` multiple times\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#22)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-06-25T13:22:33Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "c6d6d9f39949941958639356eef9f306d2b5ae20",
"body": "Project import generated by Copybara.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#21)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-06-21T20:41:13Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "556aca04ba5d79a73bbb19fc5a980832ca23a845",
"body": "Project import generated by Copybara.\r\n\r\nCloses #19.\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#20)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-06-21T18:16:29Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "dcb23fbc8a2c4a1e6f32977b2086c2a784e68262",
"body": "Project import generated by Copybara.\r\n\r\nMajor changes:\r\n\r\n- Refactor DriverImpl to use a single shared Python script for running\r\nTaskFamily methods inside the task environment\r\n- Workbench: Decompose `npm run task` into four commands, `npm run\r\n[task/agent/score/export]`\r\n\r\nCo-authored-by: Task Standard Authors <task-standard@metr.org>",
"is_bot": false,
"headline": "Sync task-standard (#18)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-06-10T21:04:20Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2bba9c132b9368a227fd88f8df23fc3c2173f1f0",
"body": "Co-authored-by: Thomas Broadley <thomas@evals.alignment.org>",
"is_bot": false,
"headline": "Add note about adaptors to README (#16)",
"author_name": "barnes-b",
"author_login": "barnes-b",
"committed_at": "2024-05-20T19:15:52Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "6294bd7ad53631aad2a324125762c7b74843ec77",
"body": null,
"is_bot": false,
"headline": "Add task definitions for GAIA and GPQA's Diamond set",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-05-15T00:29:55Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "de43ddc0f3214aeb6685212f044088dbebcf8b95",
"body": null,
"is_bot": false,
"headline": "Move pico_ctf assets zip file to Amazon S3",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-05-15T00:10:40Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "bd5c70e3ae1f4cc7a6d861ee7e779844b0300036",
"body": null,
"is_bot": false,
"headline": "0.2.3 (#15)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-05-14T05:01:17Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "df07217f75531df21c6c1f78a0e35948ad84a299",
"body": null,
"is_bot": false,
"headline": "Fix workbench submission waiting on Windows",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-05-09T21:30:14Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0db38af00a631ad0b7241ec7e09d14c0a91b66e1",
"body": null,
"is_bot": false,
"headline": "Add agentbench and swe_bench task families to examples",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-05-02T20:12:27Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "c60b33c8c01c51c9fc84ac3f535179de3c9d7fd8",
"body": null,
"is_bot": false,
"headline": "Update README.md",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-04-26T01:54:10Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "11b21ab3b167756db3e452375236df7d57565a73",
"body": "- Support custom code for task environment teardown\r\n- Support `p3` AWS instances\r\n- Fix a bug where the workbench fails to set up `iptables` rules for\r\ntask environments unless it's explicitly given root access on the\r\nmachine running Docker. The fix is to have the workbench ask for root\r\naccess using `sudo`\r\n- Other bugfixes and formatting",
"is_bot": false,
"headline": "0.2.2 (#12)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-04-23T22:54:30Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2f40b3b0326b65b1b0d4d4bfc9b71c4ca2682783",
"body": "This is reported to fix an issue with the workbench when running on\r\nWindows.",
"is_bot": false,
"headline": "Change single quotes to double quotes (#10)",
"author_name": "Ryan Bloom",
"author_login": "ryanbloom",
"committed_at": "2024-04-17T21:03:54Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "191999bdc94b0a5151190404bc1e3169b43f7938",
"body": null,
"is_bot": false,
"headline": "Allow workbench to punch holes in no-internet tasks for agent",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-04-02T01:02:48Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "24fe4e3b479279513ef573fd676f12d848270070",
"body": "TaskFamily#score may depend on the values of environment variables specified in TaskFamily#required_environment_variables. Task tests usually call TaskFamily#score. Therefore, these environment variables should be set in task tests. This PR changes the workbench to do this.",
"is_bot": false,
"headline": "Set task environment variables in task tests",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-04-02T01:02:43Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "ee95830c461c0bb749856b8057c73d4b6c167bfc",
"body": "* Add diagram of task env creation & agent execution\n\nCreating this helped me get a better sense of what happens where and which things are/aren't currently specified by the standard. Feedback welcome :)\n\n* Prettier\n\n* Update per review comments\n\n* Fix wording\n\n* Take Thomas' suggestion about shifting location of diagram\n\n---------\n\nCo-authored-by: Ted <ted.github@suzman.net>",
"is_bot": false,
"headline": "Add diagram of task env creation & agent execution (#306)",
"author_name": "Maksym Taran",
"author_login": "mtaran",
"committed_at": "2024-04-02T00:25:20Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a05927a035eaf5d6da7312de7a125c3e073e0cc1",
"body": "This is useful for debugging. I also noticed that all the other example\r\ntask families have `class Task` defined, but that is only an optional\r\npart of the spec. So this new task is a good way to make sure your\r\nimplementation of the standard covers all options.\r\n\r\n---------\r\n\r\nCo-authored-by: Thomas Broadley <buriedunderbooks@hotmail.com>",
"is_bot": false,
"headline": "Add a simple hello_world TaskFamily (#7)",
"author_name": "Art O Cathain",
"author_login": "artbristol",
"committed_at": "2024-04-02T00:22:21Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "3f039820641f4864b99d8507a4fc244140836ed8",
"body": null,
"is_bot": false,
"headline": "Update README.md",
"author_name": "Megan Kinniment",
"author_login": "MeganKW",
"committed_at": "2024-03-19T22:01:10Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "62b4c484f829559b193828c4028a5e7761a89493",
"body": null,
"is_bot": false,
"headline": "Update README.md",
"author_name": "Megan Kinniment",
"author_login": "MeganKW",
"committed_at": "2024-03-19T22:00:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "f7e87062e843d54658fcf789f918f621bd0a4053",
"body": null,
"is_bot": false,
"headline": "Make main.sh executable in example GPT-3.5 agent (#6)",
"author_name": "Art O Cathain",
"author_login": "artbristol",
"committed_at": "2024-03-19T15:51:34Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4c0d492f655ae854163657161ba7e345223f4f28",
"body": null,
"is_bot": false,
"headline": "Update LICENSE",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-03-14T22:48:39Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0be404c74f87ec5b19bb093f016b899e83b61a9f",
"body": null,
"is_bot": false,
"headline": "Create LICENSE",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-03-14T22:46:13Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4f72c49367febf9e8d75474eee38a652972af022",
"body": "In our experiments, our agents used the \"system\" role instead of the \"assistant\" role when using chat models as completion models, whereas the example baseline versions were using the \"assistant\" role. This PR makes them use the \"system\" role instead.",
"is_bot": false,
"headline": "Make legacy- and langchain-react- use system",
"author_name": "Lucas Sato",
"author_login": "satojk",
"committed_at": "2024-03-13T19:36:08Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "f6d17be1ab2eb7ad8136b0091a306b407856a8cc",
"body": null,
"is_bot": false,
"headline": "Remove -it from workbench chown",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-13T19:22:26Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "c6bd3076c5e6213c9ab0bb6f31ebacd18aa46129",
"body": "* Make sure agent has access to /home/agent/scaffolding dir\n\nThis looks like it should work!",
"is_bot": false,
"headline": "Make sure agent has access to /home/agent/scaffolding dir",
"author_name": "Maksym Taran",
"author_login": null,
"committed_at": "2024-03-13T19:17:16Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "8097b36fd36ba1b6da13e9d14ff3de6d9e8f2fdb",
"body": null,
"is_bot": false,
"headline": "Add known working Docker CLI version to workbench README",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-13T04:21:58Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "86422169566ce36e26327eda34152a743a6a10ca",
"body": null,
"is_bot": false,
"headline": "Bump standard versions to 0.2.1",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-12T23:31:07Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4b4bef6babb42067928191569b8b4c111dd50770",
"body": null,
"is_bot": false,
"headline": "Format",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-12T22:55:30Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "b0bf49c58608148d7bf9f47625fbc8fd58ab42bf",
"body": "The readme states that AWS is optional, but npm run destroy crashes without a proper AWS configuration.\r\n\r\nThis fixes that\r\n\r\nCo-authored-by: Thomas Broadley <buriedunderbooks@hotmail.com>",
"is_bot": false,
"headline": "Improve error handling in workbench if AWS is not configured (#3)",
"author_name": "Ben West",
"author_login": "Xodarap",
"committed_at": "2024-03-12T22:51:37Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0c5d120eafaf63e415092a5a1b3738fadd5d0dff",
"body": null,
"is_bot": false,
"headline": "Edit CHANGELOG",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-11T21:12:30Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "9e8c528b30878b3938e03ebe5eedd2760cc5aca5",
"body": null,
"is_bot": false,
"headline": "0.2.1",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-11T21:10:59Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "735ddfff6b3f74000acd1a79d4340e3eed76cefd",
"body": "installing dependencies from a running Python script doesn't work for me (and feels dodgy regardless). This changes the main executable to be a shell script, which I think is a more normal way of handling this.\r\n\r\nI see that other agents have pip install -r requirements.txt as part of the agent command, which would also work, but feels a bit less convenient/DRY to me.\r\n\r\nCo-authored-by: Thomas Broadley <buriedunderbooks@hotmail.com>",
"is_bot": false,
"headline": "change minimal GPT to a shell script (#4)",
"author_name": "Ben West",
"author_login": "Xodarap",
"committed_at": "2024-03-11T19:49:01Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4a0117752f2ce8eef15df5c3fbadcfb922c1a23d",
"body": "After re-running some experiments with more up-to-date versions of the\nbaseline agents, most of the \"version discrepancy notices\" are no longer\nneeded. This commit deletes those.",
"is_bot": false,
"headline": "Remove some discrepancy notices on baseline agents (#301)",
"author_name": "Lucas Sato",
"author_login": "satojk",
"committed_at": "2024-03-11T19:44:34Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "d926a25597a610a49f6e907d0de500d0ee449e61",
"body": null,
"is_bot": false,
"headline": "Mention that Packer AWS plugin is necessary",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-11T19:44:34Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "bab2cc6f973b3d7b85b7739d291e71680699e21e",
"body": null,
"is_bot": false,
"headline": "Fix vm_test_test.py",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-11T19:44:34Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "18f3414e2b639499a309bff313018a4f2c3a3780",
"body": "…ild_commands (#273)",
"is_bot": false,
"headline": "ENG-260: Build aux VM AMIs using Packer; support paths_to_copy and bu…",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-11T19:44:34Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2f250d7eddabbc4babbb1e5196513156cdcc0bc1",
"body": null,
"is_bot": false,
"headline": "Make aux VMs use public IPs by default",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-09T01:23:25Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "fdf92c769c9eb82a78e367ceef86d1765631c957",
"body": null,
"is_bot": false,
"headline": "Allow EC2 instance to be created without a public IP (#287)",
"author_name": "art-dsit",
"author_login": "art-dsit",
"committed_at": "2024-03-09T01:23:10Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "4587fb43e4cd9cb9d1c161a8f9236aa5662393db",
"body": null,
"is_bot": false,
"headline": "Reformat Python",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-08T16:06:42Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "3bfd444287247f0e10cdaf1999a9010c088c0531",
"body": "Co-authored-by: Lucas Sato <lucasjks@gmail.com>",
"is_bot": false,
"headline": "Add more example agents to the Task Standard workbench (#298)",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-08T15:53:27Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "01ae06a16f616fb21f3085dde9e05d160074a046",
"body": null,
"is_bot": false,
"headline": "Move AWS constants to constants.ts",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-08T15:53:27Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e828f3daa8f429760192cd70254f9c400b571d75",
"body": null,
"is_bot": false,
"headline": "Allow AMIs from any region (#279)",
"author_name": "art-dsit",
"author_login": "art-dsit",
"committed_at": "2024-03-08T15:53:11Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "da6abb61d35b00218b5b4e7600c3d22f9d27b567",
"body": null,
"is_bot": false,
"headline": "Snapshot aux VMs before tearing them down",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-07T15:38:55Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "d66dac9689a3a8a692869c2a3fe50c9d3b0dd8fe",
"body": null,
"is_bot": false,
"headline": "0.2.0",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-06T21:29:52Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "74ed78ffc82f122122f0b6cc94071a8b26472c97",
"body": "Two bugs. First, the agent process should be run as the agent user.\nSecond, use execFileSync and bash to run the agent in case agentCmd\ncontains shell syntax like ; and &&.\n\nExample output:\n\n(mp4) thomas@thomas-metr workbench % npm run task -- ../examples/days_since gpt-4-launch --agent-dir example-agents/minimal-gpt-3.5 --agent-cmd \"echo Hello World && whoami\"\n\n[snip]\n\n=== Setting up agent ===\nHello World\nagent",
"is_bot": false,
"headline": "Fix bugs in workbench agent execution",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-03-01T16:40:19Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "1dcc533be86c639aae4904163bdb29c80783e00f",
"body": null,
"is_bot": false,
"headline": "Update README.md",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-03-01T02:12:43Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2084c3dd597b8baca2d502415ccd48b217062e7b",
"body": null,
"is_bot": false,
"headline": "Format",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-29T22:17:36Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a7e15b9094ad2399f87f35dd15308c7a4081d93e",
"body": null,
"is_bot": false,
"headline": "Anthropic API support (#2)",
"author_name": "Lukas Petersson",
"author_login": "lukaspetersson",
"committed_at": "2024-02-29T22:05:26Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "79a3cebd679dd5b8cf2b989c02297c40a0cca1ea",
"body": null,
"is_bot": false,
"headline": "Fix workbench installation",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-29T20:33:53Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "cabe3691d44186b06a8d454b992e9a4828585b0d",
"body": null,
"is_bot": false,
"headline": "0.1.1",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-29T20:26:44Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0b0971a9934d3756e9cbf2d3a3296e6a1e171a44",
"body": null,
"is_bot": false,
"headline": "Fix typos",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-29T17:52:29Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "640bdb74d2877eb5138347e858a0693b70d28b07",
"body": null,
"is_bot": false,
"headline": "Better wording of limitations",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-02-29T17:48:32Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "5e73d7dfee51817497968a6abd302a838dea064b",
"body": null,
"is_bot": false,
"headline": "Update README.md to document limitations",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-02-29T17:47:00Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e95ce416d79410514ad6825da820062b30e3004a",
"body": null,
"is_bot": false,
"headline": "Workbench fixes (#1)",
"author_name": "Ted",
"author_login": "RoboTeddy",
"committed_at": "2024-02-21T16:05:09Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "6779051ccfe96a54e26e623ad4a684bbcb7ce122",
"body": null,
"is_bot": false,
"headline": "workbench can now run agents, with examples (thanks to @nielsrolf)",
"author_name": "Ted Suzman",
"author_login": "RoboTeddy",
"committed_at": "2024-02-13T08:27:12Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "12b8f7a0ff9b66635c04b3a5a55d5e2efc056194",
"body": null,
"is_bot": false,
"headline": "Update",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-12T16:55:42Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "f26b10164c83909d7c40c9c00881aa9f094de925",
"body": null,
"is_bot": false,
"headline": "Update README.md",
"author_name": "barnes-b",
"author_login": "barnes-b",
"committed_at": "2024-02-04T01:17:08Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "2b9485e103ca9fa9be204eceeb7d89862dd772d2",
"body": null,
"is_bot": false,
"headline": "Reintroduce accidentally reverted change",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-03T03:06:37Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a86b4fbfa21fef19d9461dc85edc73838705d334",
"body": null,
"is_bot": false,
"headline": "Task Standard launch fast follows",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-03T03:06:04Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "a7cdfce95262d2b19994f3f9230329078fde7484",
"body": null,
"is_bot": false,
"headline": "Fix post-task-start chown regression",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-02-03T00:59:45Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "77ccfbebd0b34121af39ddb2f62948c5d4db3fe7",
"body": null,
"is_bot": false,
"headline": "update readme",
"author_name": "Ted Suzman",
"author_login": "RoboTeddy",
"committed_at": "2024-01-31T22:38:53Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "0060977704e6de7bed6960ddbecf934c26b804d1",
"body": "…t an int)",
"is_bot": false,
"headline": "Update examples to conform to the standard (score returns a float, no…",
"author_name": "Ted Suzman",
"author_login": "RoboTeddy",
"committed_at": "2024-01-31T21:30:12Z",
"body_truncated": false,
"is_coding_agent": false
},
{
"oid": "e019d01e77576aa8d57c921edca7ede55d752521",
"body": null,
"is_bot": false,
"headline": "Initial commit",
"author_name": "Thomas Broadley",
"author_login": null,
"committed_at": "2024-01-31T19:42:54Z",
"body_truncated": false,
"is_coding_agent": false
}
],
"releases_count": 6,
"commits_last_year": 0,
"latest_release_at": "2024-07-31T00:57:14Z",
"latest_release_tag": "v0.3.0",
"releases_from_tags": false,
"days_since_last_push": 540,
"active_weeks_last_year": 0,
"days_since_latest_release": 728,
"mean_days_between_releases": 30.4
},
"community": {
"has_readme": true,
"has_license": true,
"has_description": true,
"has_contributing": true,
"health_percentage": 75,
"has_issue_template": false,
"has_code_of_conduct": false,
"has_pull_request_template": true
},
"ecosystem": {
"packages": []
},
"popularity": {
"forks": 37,
"stars": 186,
"watchers": 7,
"fork_history": {
"days": [
{
"date": "2024-02-07",
"count": 1
},
{
"date": "2024-02-24",
"count": 1
},
{
"date": "2024-02-29",
"count": 2
},
{
"date": "2024-03-02",
"count": 1
},
{
"date": "2024-03-03",
"count": 1
},
{
"date": "2024-03-10",
"count": 1
},
{
"date": "2024-03-11",
"count": 1
},
{
"date": "2024-03-12",
"count": 1
},
{
"date": "2024-03-18",
"count": 1
},
{
"date": "2024-03-19",
"count": 1
},
{
"date": "2024-03-23",
"count": 2
},
{
"date": "2024-03-24",
"count": 1
},
{
"date": "2024-03-29",
"count": 1
},
{
"date": "2024-04-02",
"count": 1
},
{
"date": "2024-04-03",
"count": 1
},
{
"date": "2024-04-17",
"count": 2
},
{
"date": "2024-04-29",
"count": 1
},
{
"date": "2024-05-13",
"count": 1
},
{
"date": "2024-06-12",
"count": 1
},
{
"date": "2024-07-23",
"count": 1
},
{
"date": "2024-08-01",
"count": 1
},
{
"date": "2024-08-06",
"count": 1
},
{
"date": "2024-08-13",
"count": 1
},
{
"date": "2024-08-16",
"count": 1
},
{
"date": "2024-08-17",
"count": 1
},
{
"date": "2024-12-03",
"count": 1
},
{
"date": "2024-12-04",
"count": 1
},
{
"date": "2025-01-03",
"count": 1
},
{
"date": "2025-07-29",
"count": 1
},
{
"date": "2025-08-17",
"count": 1
},
{
"date": "2025-11-04",
"count": 1
},
{
"date": "2026-04-29",
"count": 1
},
{
"date": "2026-06-20",
"count": 1
}
],
"complete": true,
"collected": 36,
"total_forks": 37
},
"star_history": null,
"open_issues_and_prs": 10
},
"ai_readiness": {
"has_nix": false,
"example_dirs": [
"examples"
],
"has_llms_txt": false,
"has_dockerfile": true,
"has_mcp_signal": false,
"bootstrap_files": [],
"api_schema_files": [],
"has_devcontainer": false,
"typecheck_configs": [
"drivers/tsconfig.json",
"workbench/tsconfig.json"
],
"toolchain_manifests": [],
"largest_source_bytes": 29920,
"source_files_sampled": 76,
"oversized_source_files": 0,
"agent_instruction_files": [],
"agent_instruction_max_bytes": null
},
"dependencies": {
"manifests": [
"drivers/package.json",
"python-package/setup.py",
"workbench/package.json"
],
"advisories": {
"error": null,
"scope": "repository_graph",
"source": "osv",
"findings": [
{
"name": "fast-xml-parser",
"direct": false,
"version": "4.2.5",
"severity": "critical",
"ecosystem": "npm",
"cvss_score": 9.3,
"advisory_ids": [
"GHSA-8gc5-j5rx-235r",
"GHSA-fj3w-jwp8-x2g3",
"GHSA-gh4j-gqv2-49f6",
"GHSA-jmr7-xgp7-cmfj",
"GHSA-jp2q-39xq-3w4g",
"GHSA-m7jm-9gc2-mpf2"
],
"fixed_version": "5.7.0",
"advisory_count": 6,
"oldest_advisory_days": 161
},
{
"name": "torch",
"direct": false,
"version": "2.1.2+cpu.cxx11.abi",
"severity": "critical",
"ecosystem": "pypi",
"cvss_score": 9.8,
"advisory_ids": [
"GHSA-3749-ghw9-m3mg",
"GHSA-53q9-r3pm-6pq6",
"GHSA-5pcm-hx3q-hm94",
"GHSA-887c-mr87-cxwp",
"GHSA-c678-jfcj-6jmf",
"GHSA-f4hp-rmr7-r7v8",
"GHSA-pg7h-5qx3-wjr3",
"GHSA-qfhq-4f3w-5fph",
"GHSA-rrmf-rvhw-rf47",
"GHSA-vgrw-7cvw-pwgx"
],
"fixed_version": "2.13.0",
"advisory_count": 28,
"oldest_advisory_days": 832
},
{
"name": "uuid",
"direct": false,
"version": "8.3.2",
"severity": "high",
"ecosystem": "npm",
"cvss_score": 7.5,
"advisory_ids": [
"GHSA-w5hq-g745-h8pq"
],
"fixed_version": "13.0.1",
"advisory_count": 1,
"oldest_advisory_days": 97
},
{
"name": "uuid",
"direct": false,
"version": "9.0.1",
"severity": "high",
"ecosystem": "npm",
"cvss_score": 7.5,
"advisory_ids": [
"GHSA-w5hq-g745-h8pq"
],
"fixed_version": "13.0.1",
"advisory_count": 1,
"oldest_advisory_days": 97
},
{
"name": "esbuild",
"direct": true,
"version": "0.21.4",
"severity": "moderate",
"ecosystem": "npm",
"cvss_score": 5.3,
"advisory_ids": [
"GHSA-67mh-4wv8-2f99"
],
"fixed_version": "0.25.0",
"advisory_count": 1,
"oldest_advisory_days": 533
},
{
"name": "zod",
"direct": true,
"version": "3.21.4",
"severity": "moderate",
"ecosystem": "npm",
"cvss_score": 5.3,
"advisory_ids": [
"GHSA-m95q-7qp3-xv42"
],
"fixed_version": "3.22.3",
"advisory_count": 1,
"oldest_advisory_days": 1034
},
{
"name": "pypdf2",
"direct": false,
"version": "3.0.1",
"severity": "moderate",
"ecosystem": "pypi",
"cvss_score": 6.2,
"advisory_ids": [
"GHSA-4vvm-4w3v-6mr8",
"PYSEC-2026-1835"
],
"fixed_version": "3.9.0",
"advisory_count": 2,
"oldest_advisory_days": 1124
},
{
"name": "@smithy/config-resolver",
"direct": false,
"version": "2.1.4",
"severity": "low",
"ecosystem": "npm",
"cvss_score": 3.7,
"advisory_ids": [
"GHSA-6475-r3vj-m8vf"
],
"fixed_version": "4.4.0",
"advisory_count": 1,
"oldest_advisory_days": 201
}
],
"collected": true,
"malicious": [],
"truncated": false,
"by_severity": {
"low": 1,
"high": 2,
"critical": 2,
"moderate": 3
},
"advisory_count": 41,
"affected_count": 8,
"assessed_count": 137,
"malicious_count": 0,
"assessed_package": null,
"unassessed_count": 22,
"direct_affected_count": 2
},
"ecosystems": [
"npm",
"pypi"
],
"dependencies": [
{
"name": "@aws-sdk/client-ec2",
"manifest": "drivers/package.json",
"ecosystem": "npm",
"version_constraint": "^3.515.0"
},
{
"name": "@types/node",
"manifest": "drivers/package.json",
"ecosystem": "npm",
"version_constraint": "^20.14.7"
},
{
"name": "json5",
"manifest": "drivers/package.json",
"ecosystem": "npm",
"version_constraint": "^2.2.3"
},
{
"name": "object-hash",
"manifest": "drivers/package.json",
"ecosystem": "npm",
"version_constraint": "^3.0.0"
},
{
"name": "zod",
"manifest": "drivers/package.json",
"ecosystem": "npm",
"version_constraint": "^3.21.4"
},
{
"name": "esbuild",
"manifest": "workbench/package.json",
"ecosystem": "npm",
"version_constraint": "^0.21.0"
},
{
"name": "esbuild-runner",
"manifest": "workbench/package.json",
"ecosystem": "npm",
"version_constraint": "^2.2.2"
},
{
"name": "zod-to-json-schema",
"manifest": "workbench/package.json",
"ecosystem": "npm",
"version_constraint": "^3.23.3"
},
{
"name": "yargs",
"manifest": "workbench/package.json",
"ecosystem": "npm",
"version_constraint": "^17.7.2"
}
],
"all_dependencies": {
"error": null,
"source": "github-sbom",
"packages": [
{
"name": "@aws-sdk/client-ec2",
"direct": true,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@types/node",
"direct": true,
"version": "20.12.12",
"ecosystem": "npm"
},
{
"name": "@types/node",
"direct": true,
"version": "20.14.7",
"ecosystem": "npm"
},
{
"name": "esbuild",
"direct": true,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "esbuild-runner",
"direct": true,
"version": "2.2.2",
"ecosystem": "npm"
},
{
"name": "object-hash",
"direct": true,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "yargs",
"direct": true,
"version": "17.7.2",
"ecosystem": "npm"
},
{
"name": "zod",
"direct": true,
"version": "3.21.4",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/crc32",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/ie11-detection",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/sha256-browser",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/sha256-js",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/supports-web-crypto",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-crypto/util",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/client-sso",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/client-sso-oidc",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/client-sts",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/core",
"direct": false,
"version": "3.513.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-env",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-http",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-ini",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-node",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-process",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-sso",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/credential-provider-web-identity",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/middleware-host-header",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/middleware-logger",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/middleware-recursion-detection",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/middleware-sdk-ec2",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/middleware-user-agent",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/region-config-resolver",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/token-providers",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/types",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-endpoints",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-format-url",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-locate-window",
"direct": false,
"version": "3.495.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-user-agent-browser",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-user-agent-node",
"direct": false,
"version": "3.515.0",
"ecosystem": "npm"
},
{
"name": "@aws-sdk/util-utf8-browser",
"direct": false,
"version": "3.259.0",
"ecosystem": "npm"
},
{
"name": "@esbuild/aix-ppc64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/android-arm",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/android-arm64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/android-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/darwin-arm64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/darwin-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/freebsd-arm64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/freebsd-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-arm",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-arm64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-ia32",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-loong64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-mips64el",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-ppc64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-riscv64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-s390x",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/linux-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/netbsd-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/openbsd-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/sunos-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/win32-arm64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/win32-ia32",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@esbuild/win32-x64",
"direct": false,
"version": "0.21.4",
"ecosystem": "npm"
},
{
"name": "@smithy/abort-controller",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/config-resolver",
"direct": false,
"version": "2.1.4",
"ecosystem": "npm"
},
{
"name": "@smithy/core",
"direct": false,
"version": "1.3.5",
"ecosystem": "npm"
},
{
"name": "@smithy/credential-provider-imds",
"direct": false,
"version": "2.2.4",
"ecosystem": "npm"
},
{
"name": "@smithy/eventstream-codec",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/fetch-http-handler",
"direct": false,
"version": "2.4.3",
"ecosystem": "npm"
},
{
"name": "@smithy/hash-node",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/invalid-dependency",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/is-array-buffer",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/middleware-content-length",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/middleware-endpoint",
"direct": false,
"version": "2.4.4",
"ecosystem": "npm"
},
{
"name": "@smithy/middleware-retry",
"direct": false,
"version": "2.1.4",
"ecosystem": "npm"
},
{
"name": "@smithy/middleware-serde",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/middleware-stack",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/node-config-provider",
"direct": false,
"version": "2.2.4",
"ecosystem": "npm"
},
{
"name": "@smithy/node-http-handler",
"direct": false,
"version": "2.4.1",
"ecosystem": "npm"
},
{
"name": "@smithy/property-provider",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/protocol-http",
"direct": false,
"version": "3.2.1",
"ecosystem": "npm"
},
{
"name": "@smithy/querystring-builder",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/querystring-parser",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/service-error-classification",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/shared-ini-file-loader",
"direct": false,
"version": "2.3.4",
"ecosystem": "npm"
},
{
"name": "@smithy/signature-v4",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/smithy-client",
"direct": false,
"version": "2.4.2",
"ecosystem": "npm"
},
{
"name": "@smithy/types",
"direct": false,
"version": "2.10.1",
"ecosystem": "npm"
},
{
"name": "@smithy/url-parser",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/util-base64",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-body-length-browser",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-body-length-node",
"direct": false,
"version": "2.2.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-buffer-from",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-config-provider",
"direct": false,
"version": "2.2.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-defaults-mode-browser",
"direct": false,
"version": "2.1.4",
"ecosystem": "npm"
},
{
"name": "@smithy/util-defaults-mode-node",
"direct": false,
"version": "2.2.3",
"ecosystem": "npm"
},
{
"name": "@smithy/util-endpoints",
"direct": false,
"version": "1.1.4",
"ecosystem": "npm"
},
{
"name": "@smithy/util-hex-encoding",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-middleware",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/util-retry",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/util-stream",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@smithy/util-uri-escape",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-utf8",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "@smithy/util-waiter",
"direct": false,
"version": "2.1.3",
"ecosystem": "npm"
},
{
"name": "@types/object-hash",
"direct": false,
"version": "3.0.6",
"ecosystem": "npm"
},
{
"name": "@types/yargs",
"direct": false,
"version": "17.0.32",
"ecosystem": "npm"
},
{
"name": "@types/yargs-parser",
"direct": false,
"version": "21.0.3",
"ecosystem": "npm"
},
{
"name": "ansi-regex",
"direct": false,
"version": "5.0.1",
"ecosystem": "npm"
},
{
"name": "ansi-styles",
"direct": false,
"version": "4.3.0",
"ecosystem": "npm"
},
{
"name": "bowser",
"direct": false,
"version": "2.11.0",
"ecosystem": "npm"
},
{
"name": "buffer-from",
"direct": false,
"version": "1.1.2",
"ecosystem": "npm"
},
{
"name": "cliui",
"direct": false,
"version": "8.0.1",
"ecosystem": "npm"
},
{
"name": "color-convert",
"direct": false,
"version": "2.0.1",
"ecosystem": "npm"
},
{
"name": "color-name",
"direct": false,
"version": "1.1.4",
"ecosystem": "npm"
},
{
"name": "emoji-regex",
"direct": false,
"version": "8.0.0",
"ecosystem": "npm"
},
{
"name": "escalade",
"direct": false,
"version": "3.1.2",
"ecosystem": "npm"
},
{
"name": "fast-xml-parser",
"direct": false,
"version": "4.2.5",
"ecosystem": "npm"
},
{
"name": "get-caller-file",
"direct": false,
"version": "2.0.5",
"ecosystem": "npm"
},
{
"name": "is-fullwidth-code-point",
"direct": false,
"version": "3.0.0",
"ecosystem": "npm"
},
{
"name": "prettier-no-jsx-parens",
"direct": false,
"version": "2.8.4",
"ecosystem": "npm"
},
{
"name": "require-directory",
"direct": false,
"version": "2.1.1",
"ecosystem": "npm"
},
{
"name": "source-map",
"direct": false,
"version": "0.6.1",
"ecosystem": "npm"
},
{
"name": "source-map-support",
"direct": false,
"version": "0.5.21",
"ecosystem": "npm"
},
{
"name": "string-width",
"direct": false,
"version": "4.2.3",
"ecosystem": "npm"
},
{
"name": "strip-ansi",
"direct": false,
"version": "6.0.1",
"ecosystem": "npm"
},
{
"name": "strnum",
"direct": false,
"version": "1.0.5",
"ecosystem": "npm"
},
{
"name": "tslib",
"direct": false,
"version": "1.14.1",
"ecosystem": "npm"
},
{
"name": "tslib",
"direct": false,
"version": "2.4.0",
"ecosystem": "npm"
},
{
"name": "tslib",
"direct": false,
"version": "2.6.2",
"ecosystem": "npm"
},
{
"name": "undici-types",
"direct": false,
"version": "5.26.5",
"ecosystem": "npm"
},
{
"name": "uuid",
"direct": false,
"version": "8.3.2",
"ecosystem": "npm"
},
{
"name": "uuid",
"direct": false,
"version": "9.0.1",
"ecosystem": "npm"
},
{
"name": "wrap-ansi",
"direct": false,
"version": "7.0.0",
"ecosystem": "npm"
},
{
"name": "y18n",
"direct": false,
"version": "5.0.8",
"ecosystem": "npm"
},
{
"name": "yargs-parser",
"direct": false,
"version": "21.1.1",
"ecosystem": "npm"
},
{
"name": "anthropic",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "beautifulsoup4",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "bokeh",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "cachetools",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "datasets",
"direct": false,
"version": "2.15.0",
"ecosystem": "pypi"
},
{
"name": "ffmpeg-python",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "ipython",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "langchain",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "matplotlib",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "numpy",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "openai",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "pandas",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "plotly",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "pydantic",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "pypdf2",
"direct": false,
"version": "3.0.1",
"ecosystem": "pypi"
},
{
"name": "pytest",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "python-dotenv",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "scikit-learn",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "sortedcontainers",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "speechrecognition",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "tabulate",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "textract",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "tiktoken",
"direct": false,
"version": null,
"ecosystem": "pypi"
},
{
"name": "torch",
"direct": false,
"version": "2.1.2+cpu.cxx11.abi",
"ecosystem": "pypi"
},
{
"name": "xgboost",
"direct": false,
"version": null,
"ecosystem": "pypi"
}
],
"collected": true,
"truncated": false,
"total_count": 159,
"direct_count": 8,
"indirect_count": 151
}
},
"maintainership": {
"issues": {
"open_prs": 2,
"merged_prs": 21,
"open_issues": 8,
"closed_ratio": 0.385,
"closed_issues": 5,
"closed_unmerged_prs": 4
},
"bus_factor": 2,
"bot_contributors": 0,
"top_contributors": [
{
"type": "User",
"login": "RoboTeddy",
"commits": 10,
"avatar_url": "https://avatars.githubusercontent.com/u/172271?v=4"
},
{
"type": "User",
"login": "mtaran",
"commits": 8,
"avatar_url": "https://avatars.githubusercontent.com/u/65346?v=4"
},
{
"type": "User",
"login": "art-dsit",
"commits": 3,
"avatar_url": "https://avatars.githubusercontent.com/u/153507562?v=4"
},
{
"type": "User",
"login": "artbristol",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/925073?v=4"
},
{
"type": "User",
"login": "Xodarap",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/221511?v=4"
},
{
"type": "User",
"login": "satojk",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/21082800?v=4"
},
{
"type": "User",
"login": "MeganKW",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/70336425?v=4"
},
{
"type": "User",
"login": "tbroadley",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/8731922?v=4"
},
{
"type": "User",
"login": "barnes-b",
"commits": 2,
"avatar_url": "https://avatars.githubusercontent.com/u/13219333?v=4"
},
{
"type": "User",
"login": "lukaspetersson",
"commits": 1,
"avatar_url": "https://avatars.githubusercontent.com/u/42323981?v=4"
}
],
"contributors_sampled": 12,
"top_contributor_share": 0.278
},
"quality_signals": {
"has_ci": false,
"has_tests": true,
"ci_workflows": [],
"has_docs_dir": false,
"linter_configs": [],
"has_editorconfig": false,
"has_linter_config": false,
"has_precommit_config": false
},
"security_signals": {
"lockfiles": [
"package-lock.json"
],
"scorecard": null,
"has_codeql_workflow": false,
"has_security_policy": false,
"has_dependabot_config": false
},
"contribution_flow": {
"collected": true,
"ci_last_run_at": "2026-07-23T18:02:02Z",
"oldest_open_prs": [
{
"number": 9,
"created_at": "2024-04-02T16:39:37Z",
"last_comment_at": null,
"last_comment_author": null
},
{
"number": 13,
"created_at": "2024-04-29T05:31:00Z",
"last_comment_at": null,
"last_comment_author": null
}
],
"last_merged_pr_at": "2024-10-30T22:47:29Z",
"ci_last_conclusion": "SUCCESS",
"oldest_open_issues": [
{
"number": 8,
"created_at": "2024-04-01T12:25:56Z",
"last_comment_at": "2024-06-24T16:26:44Z",
"last_comment_author": "sjawhar"
},
{
"number": 14,
"created_at": "2024-05-13T10:05:01Z",
"last_comment_at": "2024-05-22T15:14:16Z",
"last_comment_author": "art-dsit"
},
{
"number": 17,
"created_at": "2024-05-31T14:46:54Z",
"last_comment_at": "2024-05-31T18:07:14Z",
"last_comment_author": "tbroadley"
},
{
"number": 33,
"created_at": "2024-09-24T21:28:11Z",
"last_comment_at": null,
"last_comment_author": null
},
{
"number": 35,
"created_at": "2024-10-09T14:36:52Z",
"last_comment_at": "2024-10-13T13:26:50Z",
"last_comment_author": "hibukki"
},
{
"number": 38,
"created_at": "2024-12-24T18:58:42Z",
"last_comment_at": "2025-01-02T19:26:46Z",
"last_comment_author": "tbroadley"
},
{
"number": 40,
"created_at": "2025-12-23T12:41:41Z",
"last_comment_at": "2026-03-29T07:18:31Z",
"last_comment_author": "agent-morrow"
},
{
"number": 41,
"created_at": "2026-05-10T22:22:03Z",
"last_comment_at": "2026-05-11T15:52:06Z",
"last_comment_author": "tbroadley"
}
]
}
},
"config": {
"disabled_metrics": [],
"disabled_categories": [],
"disabled_components": {}
},
"source": {
"url": "https://github.com/METR/task-standard",
"host": "github.com",
"name": "task-standard",
"owner": "METR"
},
"metrics": {
"overall": {
"key": "overall",
"band": "weak",
"name": "Overall health",
"note": "The weighted overall 45 is calibrated to 42 on the published index scale (record calibration 2026-08-02). Abandonment Policy applies a 85% multiplier to weighted overall health.",
"notes": [
{
"code": "overall_calibration",
"params": {
"raw": 45,
"calibrated": 42,
"calibration": "2026-08-02"
}
},
{
"code": "abandonment_overall_adjustment",
"params": {
"cap": 0,
"pct": 85
}
}
],
"value": 36,
"inputs": {
"security": 33,
"vitality": 28,
"community": 65,
"governance": 63,
"calibration": "2026-08-02",
"engineering": 34,
"ai_readiness": 47,
"abandonment_cap": null,
"abandonment_state": "at_risk",
"weighted_overall_raw": 45,
"abandonment_multiplier": 85,
"weighted_overall_before_abandonment": 42,
"overall_after_abandonment_multiplier": 36
},
"components": []
},
"categories": [
{
"key": "vitality",
"band": "at_risk",
"name": "Vitality",
"value": 28,
"weight": 0.21,
"metrics": [
{
"key": "development_activity",
"band": "critical",
"name": "Development activity",
"note": null,
"notes": [],
"value": 1,
"inputs": {
"commits_last_year": 0,
"human_commit_share": 1,
"days_since_last_push": 540,
"active_weeks_last_year": 0
},
"components": [
{
"key": "push_recency",
"name": "Push recency",
"detail": "last push 540 days ago",
"points": 0,
"status": "missed",
"details": [
{
"code": "push_recency",
"params": {
"days": 540
}
}
],
"max_points": 36
},
{
"key": "commit_cadence",
"name": "Commit cadence",
"detail": "0/52 weeks with commits",
"points": 0,
"status": "missed",
"details": [
{
"code": "commit_cadence_weeks",
"params": {
"weeks": 0
}
}
],
"max_points": 36
},
{
"key": "commit_volume",
"name": "Commit volume",
"detail": "0 commits in the last year",
"points": 0,
"status": "missed",
"details": [
{
"code": "commits_last_year",
"params": {
"count": 0
}
}
],
"max_points": 18
},
{
"key": "openssf_scorecard_maintained",
"name": "OpenSSF Scorecard: Maintained",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "release_discipline",
"band": "good",
"name": "Release discipline",
"note": null,
"notes": [],
"value": 68,
"inputs": {
"releases_count": 6,
"latest_release_tag": "v0.3.0",
"releases_from_tags": false,
"days_since_latest_release": 728,
"mean_days_between_releases": 30.4
},
"components": [
{
"key": "ships_releases",
"name": "Ships releases",
"detail": "6 releases published",
"points": 27,
"status": "met",
"details": [
{
"code": "releases_published",
"params": {
"count": 6
}
}
],
"max_points": 27
},
{
"key": "release_recency",
"name": "Release recency",
"detail": "latest release 728 days ago",
"points": 7.2,
"status": "partial",
"details": [
{
"code": "release_recency",
"params": {
"days": 728
}
}
],
"max_points": 36
},
{
"key": "release_cadence",
"name": "Release cadence",
"detail": "a release every ~30.4 days",
"points": 27,
"status": "met",
"details": [
{
"code": "release_cadence",
"params": {
"gap": 30.4
}
}
],
"max_points": 27
},
{
"key": "openssf_scorecard_signed_releases",
"name": "OpenSSF Scorecard: Signed-Releases",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "abandonment",
"band": "excellent",
"name": "Abandonment",
"note": null,
"notes": [],
"value": 85,
"inputs": {
"cap": null,
"state": "at_risk",
"guards": [
"maintainer_replying"
],
"signals": [
"unfixed_advisory",
"release_stall"
],
"red_flag": true,
"multiplier_pct": 85,
"declared_reason": null,
"unverified_reason": null,
"unanswered_open_prs": 2,
"unanswered_open_issues": 3,
"days_since_last_merged_pr": 641,
"days_since_last_human_commit": 545,
"days_since_last_human_commit_is_floor": false
},
"components": [
{
"key": "project_is_still_maintained",
"name": "Project is still maintained",
"detail": "no human commit for 545 days; 2 unmet obligation(s): a fixed advisory left unapplied, releases stalled",
"points": 85,
"status": "partial",
"details": [
{
"code": "abandonment_flagged",
"params": {
"days": 545,
"count": 2,
"signals": "a fixed advisory left unapplied, releases stalled"
}
}
],
"max_points": 100
}
]
}
],
"description": "Is the project alive — is code being written and are releases shipping?"
},
{
"key": "community",
"band": "good",
"name": "Community & Adoption",
"value": 65,
"weight": 0.17,
"metrics": [
{
"key": "popularity",
"band": "moderate",
"name": "Popularity & adoption",
"note": null,
"notes": [],
"value": 54,
"inputs": {
"forks": 37,
"stars": 186,
"watchers": 7,
"growth_state": "unverified",
"growth_factor_pct": 100,
"growth_unverified_reason": "no_history"
},
"components": [
{
"key": "stars",
"name": "Stars",
"detail": "186 stars",
"points": 36.8,
"status": "partial",
"details": [
{
"code": "stars",
"params": {
"count": 186
}
}
],
"max_points": 60
},
{
"key": "forks",
"name": "Forks",
"detail": "37 forks",
"points": 13,
"status": "partial",
"details": [
{
"code": "forks",
"params": {
"count": 37
}
}
],
"max_points": 25
},
{
"key": "watchers",
"name": "Watchers",
"detail": "7 watchers",
"points": 4.3,
"status": "partial",
"details": [
{
"code": "watchers",
"params": {
"count": 7
}
}
],
"max_points": 15
}
]
},
{
"key": "community_health",
"band": "good",
"name": "Community health",
"note": null,
"notes": [],
"value": 77,
"inputs": {
"has_readme": true,
"has_license": true,
"readme_badges": null,
"has_contributing": true,
"has_issue_template": false,
"has_code_of_conduct": false,
"readme_badge_services": [],
"has_pull_request_template": true
},
"components": [
{
"key": "readme",
"name": "README",
"detail": null,
"points": 22.5,
"status": "met",
"details": [],
"max_points": 22.5
},
{
"key": "license",
"name": "License",
"detail": "recognized license (MIT)",
"points": 22.5,
"status": "met",
"details": [
{
"code": "license_standard",
"params": {}
},
{
"code": "license_spdx",
"params": {
"spdx": "MIT"
}
}
],
"max_points": 22.5
},
{
"key": "contributing_guide",
"name": "CONTRIBUTING guide",
"detail": null,
"points": 18,
"status": "met",
"details": [],
"max_points": 18
},
{
"key": "code_of_conduct",
"name": "Code of conduct",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 13.5
},
{
"key": "issue_template",
"name": "Issue template",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 7.2
},
{
"key": "pr_template",
"name": "PR template",
"detail": null,
"points": 6.3,
"status": "met",
"details": [],
"max_points": 6.3
}
]
}
],
"description": "Does the project have users, downloads, attention, and a welcoming setup for contributors?"
},
{
"key": "governance",
"band": "moderate",
"name": "Sustainability & Governance",
"value": 63,
"weight": 0.23,
"metrics": [
{
"key": "maintainer_resilience",
"band": "moderate",
"name": "Maintainer resilience (bus factor)",
"note": null,
"notes": [],
"value": 61,
"inputs": {
"bus_factor": 2,
"contributors_sampled": 12,
"top_contributor_share": 0.278
},
"components": [
{
"key": "bus_factor",
"name": "Bus factor",
"detail": "2 contributor(s) cover half of all commits",
"points": 25.2,
"status": "partial",
"details": [
{
"code": "bus_factor",
"params": {
"count": 2
}
}
],
"max_points": 54
},
{
"key": "commit_distribution",
"name": "Commit distribution",
"detail": "top contributor authored 28% of commits",
"points": 16.2,
"status": "partial",
"details": [
{
"code": "top_contributor_share",
"params": {
"share": 28
}
}
],
"max_points": 22.5
},
{
"key": "contributor_breadth",
"name": "Contributor breadth",
"detail": "12 contributors",
"points": 13.5,
"status": "met",
"details": [
{
"code": "contributors_sampled",
"params": {
"count": 12
}
}
],
"max_points": 13.5
},
{
"key": "openssf_scorecard_contributors",
"name": "OpenSSF Scorecard: Contributors",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "responsiveness",
"band": "moderate",
"name": "Issue & PR responsiveness",
"note": "Excluded from scoring (no data or not applicable): Newcomer PR acceptance. Remaining weights renormalized.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"newcomer_pr_acceptance"
]
}
},
{
"code": "weights_renormalized",
"params": {}
}
],
"value": 58,
"inputs": {
"merged_prs": 21,
"open_issues": 8,
"closed_issues": 5,
"prs_merged_7d": null,
"prs_decided_7d": null,
"prs_merged_30d": null,
"prs_decided_30d": null,
"issue_closed_ratio": 0.385,
"closed_unmerged_prs": 4,
"first_time_authors_30d": null,
"first_time_prs_merged_30d": null,
"first_time_prs_decided_30d": null
},
"components": [
{
"key": "issue_resolution",
"name": "Issue resolution",
"detail": "38% of issues closed",
"points": 16.2,
"status": "partial",
"details": [
{
"code": "issues_closed_share",
"params": {
"share": 38
}
}
],
"max_points": 42
},
{
"key": "pr_acceptance",
"name": "PR acceptance",
"detail": "21/25 decided PRs merged",
"points": 25.2,
"status": "partial",
"details": [
{
"code": "decided_prs_merged",
"params": {
"merged": 21,
"decided": 25
}
}
],
"max_points": 30
},
{
"key": "newcomer_pr_acceptance",
"name": "Newcomer PR acceptance",
"detail": "no first-time contributor's PR decided in 30d",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_newcomer_prs",
"params": {
"days": 30
}
}
],
"max_points": 13
},
{
"key": "openssf_scorecard_code_review",
"name": "OpenSSF Scorecard: Code-Review",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 15
}
]
},
{
"key": "stewardship",
"band": "good",
"name": "Ownership & stewardship",
"note": null,
"notes": [],
"value": 70,
"inputs": {
"followers": 452,
"owner_type": "Organization",
"is_verified": null,
"owner_login": "METR",
"public_repos": 62,
"account_age_days": 1387
},
"components": [
{
"key": "ownership_backing",
"name": "Ownership backing",
"detail": "organization-owned",
"points": 30,
"status": "met",
"details": [
{
"code": "owner_organization",
"params": {}
}
],
"max_points": 30
},
{
"key": "verified_domain",
"name": "Verified domain",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 20
},
{
"key": "owner_reach",
"name": "Owner reach",
"detail": "452 followers of METR",
"points": 19.1,
"status": "partial",
"details": [
{
"code": "owner_followers",
"params": {
"count": 452,
"login": "METR"
}
}
],
"max_points": 25
},
{
"key": "track_record",
"name": "Track record",
"detail": "62 public repos, account ~3 yr old",
"points": 20.6,
"status": "partial",
"details": [
{
"code": "public_repos",
"params": {
"count": 62
}
},
{
"code": "account_age_years",
"params": {
"years": 3
}
}
],
"max_points": 25
}
]
}
],
"description": "Will the project survive its people — bus factor, responsiveness, who backs it, and package upkeep?"
},
{
"key": "engineering",
"band": "at_risk",
"name": "Engineering Quality",
"value": 34,
"weight": 0.19,
"metrics": [
{
"key": "engineering_practices",
"band": "at_risk",
"name": "Engineering practices",
"note": null,
"notes": [],
"value": 30,
"inputs": {
"has_ci": false,
"has_tests": true,
"has_editorconfig": false,
"has_linter_config": false,
"has_precommit_config": false
},
"components": [
{
"key": "ci_workflows",
"name": "CI workflows",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 24
},
{
"key": "tests_present",
"name": "Tests present",
"detail": null,
"points": 24,
"status": "met",
"details": [],
"max_points": 24
},
{
"key": "linter_config",
"name": "Linter config",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 16
},
{
"key": "pre_commit_hooks",
"name": "Pre-commit hooks",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 9.6
},
{
"key": "editorconfig",
"name": ".editorconfig",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 6.4
},
{
"key": "openssf_scorecard_ci_tests",
"name": "OpenSSF Scorecard: CI-Tests",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 20
}
]
},
{
"key": "documentation",
"band": "weak",
"name": "Documentation",
"note": null,
"notes": [],
"value": 40,
"inputs": {
"topics": [],
"has_wiki": false,
"homepage": null,
"has_readme": true,
"has_docs_dir": false,
"has_description": true
},
"components": [
{
"key": "readme",
"name": "README",
"detail": null,
"points": 30,
"status": "met",
"details": [],
"max_points": 30
},
{
"key": "documentation_directory",
"name": "Documentation directory",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 25
},
{
"key": "documentation_homepage_site",
"name": "Documentation / homepage site",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 15
},
{
"key": "repository_description",
"name": "Repository description",
"detail": null,
"points": 10,
"status": "met",
"details": [],
"max_points": 10
},
{
"key": "topics",
"name": "Topics",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 10
},
{
"key": "wiki",
"name": "Wiki",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 10
}
]
}
],
"description": "Are baseline engineering and documentation practices in place?"
},
{
"key": "security",
"band": "at_risk",
"name": "Security",
"value": 33,
"weight": 0.16,
"metrics": [
{
"key": "security_posture",
"band": "at_risk",
"name": "Security posture",
"note": null,
"notes": [],
"value": 25,
"inputs": {
"source": "file_signals",
"lockfiles": [
"package-lock.json"
],
"manifests": [
"drivers/package.json",
"python-package/setup.py",
"workbench/package.json"
],
"has_codeql_workflow": false,
"has_security_policy": false,
"has_dependabot_config": false
},
"components": [
{
"key": "security_policy_security_md",
"name": "Security policy (SECURITY.md)",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 30
},
{
"key": "dependabot_config",
"name": "Dependabot config",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 25
},
{
"key": "dependency_lockfiles",
"name": "Dependency lockfiles",
"detail": "package-lock.json",
"points": 25,
"status": "met",
"details": [
{
"code": "file_list",
"params": {
"files": "package-lock.json"
}
}
],
"max_points": 25
},
{
"key": "codeql_workflow",
"name": "CodeQL workflow",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 20
}
]
},
{
"key": "dependency_advisories",
"band": "good",
"name": "Dependency advisories",
"note": "Excluded from scoring (no data or not applicable): Indirect dependencies free of known advisories. Remaining weights renormalized. Matched 137 resolved dependencies against OSV; 22 could not be assessed (no resolved version, an unsupported ecosystem, or beyond the reported package list). This repository publishes no package the index resolves, so the repository dependency graph was assessed instead. That graph mixes development and test pins with shipped dependencies, so only the declared runtime dependencies are scored; transitive findings are reported as context and excluded from the score. Reachability is not analyzed.",
"notes": [
{
"code": "excluded_no_data",
"params": {
"components": [
"indirect_dependencies_free_of_known_advisories"
]
}
},
{
"code": "weights_renormalized",
"params": {}
},
{
"code": "advisories_scope_repository",
"params": {
"assessed": 137
}
},
{
"code": "advisories_unassessed",
"params": {
"count": 22
}
},
{
"code": "advisories_repo_graph_caveat",
"params": {}
},
{
"code": "advisories_reachability",
"params": {}
}
],
"value": 66,
"inputs": {
"source": "osv",
"advisories": 41,
"affected_packages": 8,
"assessed_packages": 137,
"unassessed_packages": 22,
"affected_by_severity": "critical 2, high 2, moderate 3, low 1",
"direct_affected_packages": 2
},
"components": [
{
"key": "direct_dependencies_free_of_known_advisories",
"name": "Direct dependencies free of known advisories",
"detail": "2 affected: esbuild 0.21.4 (moderate 5.3), zod 3.21.4 (moderate 5.3)",
"points": 17.8,
"status": "partial",
"details": [
{
"code": "advisories_affected",
"params": {
"count": 2,
"packages": "esbuild 0.21.4 (moderate 5.3), zod 3.21.4 (moderate 5.3)"
}
}
],
"max_points": 35
},
{
"key": "indirect_dependencies_free_of_known_advisories",
"name": "Indirect dependencies free of known advisories",
"detail": "transitive set not separable from development and test dependencies in this scope",
"points": 0,
"status": "excluded",
"details": [
{
"code": "advisories_scope_not_separable",
"params": {}
}
],
"max_points": 25
},
{
"key": "no_advisories_left_outstanding",
"name": "No advisories left outstanding",
"detail": "2 advisory-carrying package(s) unaddressed past 90 days; oldest published 1034 days ago",
"points": 31.6,
"status": "partial",
"details": [
{
"code": "advisories_stale",
"params": {
"days": 90,
"count": 2,
"oldest": 1034
}
}
],
"max_points": 40
}
]
},
{
"key": "malicious_dependencies",
"band": "exceptional",
"name": "Malicious dependencies",
"note": null,
"notes": [],
"value": 100,
"inputs": {
"source": "osv",
"meaning": "reported as a malicious package by the OpenSSF corpus; the remedy is removal or moving off the compromised name, never an upgrade of the same artifact. Versions the registry has since pulled are listed but not scored",
"packages": [],
"red_flag": false,
"assessed_packages": 137,
"malicious_packages": 0,
"direct_malicious_packages": 0,
"withdrawn_malicious_packages": 0,
"installable_malicious_packages": 0
},
"components": [
{
"key": "no_dependency_reported_as_a_malicious_package",
"name": "No dependency reported as a malicious package",
"detail": "no dependency is reported as a malicious package",
"points": 100,
"status": "met",
"details": [
{
"code": "no_malicious_dependencies",
"params": {}
}
],
"max_points": 100
}
]
},
{
"key": "high_risk_jurisdiction_exposure",
"band": "exceptional",
"name": "High-Risk Jurisdiction Exposure",
"note": "Only high-confidence self-published location evidence affects this multiplier. Ambiguous matches are review-only; country evidence is not proof of nationality, citizenship, legal registration, malicious intent, or sanctions status.",
"notes": [
{
"code": "jurisdiction_evidence_limits",
"params": {}
}
],
"value": 100,
"inputs": {
"meaning": "self-published location evidence; not nationality or citizenship",
"red_flag": false,
"exposures": [],
"policy_countries": [
"Russia",
"Iran",
"North Korea"
],
"commit_weight_rule": {
"min_commits": 50,
"min_commit_share": 0.1
},
"review_only_matches": 0,
"below_threshold_exposures": [],
"assessed_self_published_locations": 4
},
"components": [
{
"key": "policy_exposure_multiplier",
"name": "Policy exposure multiplier",
"detail": "no confirmed policy-scope location match",
"points": 100,
"status": "met",
"details": [
{
"code": "jurisdiction_no_match",
"params": {}
}
],
"max_points": 100
}
]
}
],
"description": "Are visible security and supply-chain practices strong, with no malicious dependency and no unresolved high-risk jurisdiction exposure?"
},
{
"key": "ai_readiness",
"band": "weak",
"name": "AI Readiness",
"value": 47,
"weight": 0.04,
"metrics": [
{
"key": "ai_agent_context",
"band": "at_risk",
"name": "Agent context & guidance",
"note": null,
"notes": [],
"value": 23,
"inputs": {
"has_llms_txt": false,
"legible_history_share": 0.425,
"agent_instruction_files": [],
"agent_instruction_max_bytes": null
},
"components": [
{
"key": "agent_instructions",
"name": "Agent instructions",
"detail": "no CLAUDE.md / AGENTS.md / editor rules",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_agent_instructions",
"params": {}
}
],
"max_points": 45
},
{
"key": "machine_readable_docs_llms_txt",
"name": "Machine-readable docs (llms.txt)",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 15
},
{
"key": "legible_commit_history",
"name": "Legible commit history",
"detail": "31 of 73 human commits state their intent (structured subject or explanatory body)",
"points": 22.6,
"status": "partial",
"details": [
{
"code": "legible_history",
"params": {
"legible": 31,
"sampled": 73
}
}
],
"max_points": 40
}
]
},
{
"key": "ai_verify_loop",
"band": "weak",
"name": "Verify loop (build / test / typecheck)",
"note": null,
"notes": [],
"value": 48,
"inputs": {
"has_nix": false,
"has_tests": true,
"lockfiles": [
"package-lock.json"
],
"has_dockerfile": true,
"typed_language": true,
"bootstrap_files": [],
"has_devcontainer": false,
"has_linter_config": false,
"typecheck_configs": [
"drivers/tsconfig.json",
"workbench/tsconfig.json"
],
"agent_commit_share": 0,
"toolchain_manifests": [],
"dependency_bot_commit_share": 0
},
"components": [
{
"key": "one_command_bootstrap",
"name": "One-command bootstrap",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 18
},
{
"key": "automated_tests",
"name": "Automated tests",
"detail": null,
"points": 22,
"status": "met",
"details": [],
"max_points": 22
},
{
"key": "lint_format_config",
"name": "Lint / format config",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 11
},
{
"key": "static_type_checking",
"name": "Static type checking",
"detail": "drivers/tsconfig.json, workbench/tsconfig.json",
"points": 11,
"status": "met",
"details": [
{
"code": "file_list",
"params": {
"files": "drivers/tsconfig.json, workbench/tsconfig.json"
}
}
],
"max_points": 11
},
{
"key": "reproducible_environment",
"name": "Reproducible environment",
"detail": "Dockerfile, lockfile",
"points": 10,
"status": "met",
"details": [
{
"code": "file_list",
"params": {
"files": "Dockerfile, lockfile"
}
}
],
"max_points": 10
},
{
"key": "demonstrated_agent_practice",
"name": "Demonstrated agent practice",
"detail": "no agent-authored commits among the last 73",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_agent_authored_commits",
"params": {
"sampled": 73
}
}
],
"max_points": 10
},
{
"key": "automated_maintenance",
"name": "Automated maintenance",
"detail": "no automated dependency updates observed",
"points": 0,
"status": "missed",
"details": [
{
"code": "no_dependency_automation",
"params": {}
}
],
"max_points": 8
},
{
"key": "openssf_scorecard_pinned_dependencies",
"name": "OpenSSF Scorecard: Pinned-Dependencies",
"detail": "OpenSSF Scorecard unavailable",
"points": 0,
"status": "excluded",
"details": [
{
"code": "no_data",
"params": {}
}
],
"max_points": 10
}
]
},
{
"key": "ai_code_legibility",
"band": "exceptional",
"name": "Code legibility for models",
"note": null,
"notes": [],
"value": 100,
"inputs": {
"primary_language": "TypeScript",
"largest_source_bytes": 29920,
"source_files_sampled": 76,
"oversized_source_files": 0
},
"components": [
{
"key": "type_checkable_code",
"name": "Type-checkable code",
"detail": "TypeScript (statically typed)",
"points": 45,
"status": "met",
"details": [
{
"code": "statically_typed_language",
"params": {
"language": "TypeScript"
}
}
],
"max_points": 45
},
{
"key": "manageable_file_sizes",
"name": "Manageable file sizes",
"detail": "0/76 source files over 60KB",
"points": 55,
"status": "met",
"details": [
{
"code": "oversized_source_files",
"params": {
"kb": 60,
"sampled": 76,
"oversized": 0
}
}
],
"max_points": 55
}
]
},
{
"key": "ai_interfaces",
"band": "weak",
"name": "Machine-readable interfaces",
"note": null,
"notes": [],
"value": 40,
"inputs": {
"example_dirs": [
"examples"
],
"has_mcp_signal": false,
"api_schema_files": []
},
"components": [
{
"key": "api_schema_openapi_graphql_proto",
"name": "API schema (OpenAPI/GraphQL/proto)",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 40
},
{
"key": "mcp_server",
"name": "MCP server",
"detail": null,
"points": 0,
"status": "missed",
"details": [],
"max_points": 20
},
{
"key": "runnable_examples",
"name": "Runnable examples",
"detail": "examples",
"points": 40,
"status": "met",
"details": [
{
"code": "file_list",
"params": {
"files": "examples"
}
}
],
"max_points": 40
}
]
}
],
"description": "How well is the repo equipped to be developed and maintained with AI coding agents? Carries a deliberately small weight: agent tooling is a real maintenance signal, but its absence must never gate the top of the scale (calibration saturates at raw 91, so 100/100 remains reachable with AI Readiness at zero)."
}
],
"classification": {
"labels": [
"cli"
],
"scores": {
"cli": 4
},
"primary": "cli",
"evidence": [
{
"tier": "dependencies",
"label": "cli",
"source": "dep:yargs",
"weight": 4
}
],
"artifacts": [],
"confidence": "low",
"host_extension": false,
"runs_as_process": true,
"consumed_by_code": false
},
"metrics_version": "2.3.1"
},
"warnings": [
"Star history unavailable: GitHub GraphQL error: Resource not accessible by personal access token",
"Could not fetch npm package 'task-standard-drivers' from its registry",
"Could not fetch npm package 'task-standard-workbench' from its registry",
"Could not fetch pypi package 'metr-task-standard' from its registry",
"OpenSSF Scorecard did not return a usable result (2026/07/29 01:31:49 Warning: PATs stored in env variables GITHUB_AUTH_TOKEN and GITHUB_TOKEN differ. Scorecard will use the former.); skipping Scorecard checks"
],
"report_type": "repository",
"generated_at": "2026-07-29T01:32:28.762326Z",
"schema_version": "0.27.0",
"badge_url": "https://raw.githubusercontent.com/inspect-software/badges/main/v1/m/METR/task-standard.svg",
"full_name": "METR/task-standard",
"license_state": "standard",
"license_spdx": "MIT"
}Scores are signals, not warranties. They reflect publicly visible practices on GitHub — not a code audit, and not a security guarantee.
Missing data is excluded and weights renormalized, never scored as zero. Methodology is versioned and open: metrics v2.3.1, schema v0.27.0 — full methodology · metrics wiki.
How one result sits in the wider record: aggregate statistics.