AI verify loop — inspect.software wiki

AI verify loop measures whether an AI coding agent can set up the project, run it, and verify its own change without human help. This is the crux of autonomous agent work — an agent that can check its work compounds; one that cannot merely generates plausible text — and it carries the heaviest weight in the AI Readiness badge.

Category: AI Readiness (40% within category)
Weight in overall index: 0% — part of the independent AI Readiness badge
Metric key: ai_verify_loop

How the value is computed

Component	Weight	Evidence
One-command bootstrap	18	Makefile, Taskfile, justfile, mise, or noxfile earns full credit; a toolchain that defines the command itself (`Cargo.toml`, `go.mod`, `mix.exs`, Maven, Gradle, `.csproj`) earns most of it
Automated tests	22	a test suite the agent can run to self-check (shared with engineering practices)
Lint / format config	11	shared with the engineering linter signal
Static type checking	11	a statically typed language, or a type-check configuration (mypy, pyright, tsconfig, `py.typed`)
Reproducible environment	10	devcontainer, Dockerfile, Nix, or a dependency lockfile
Demonstrated agent practice	10	share of recent commits authored or co-authored by a coding agent; full marks from 5% upward
Automated maintenance	8	dependency-update bot commits observed in the sample; a configuration with nothing observed earns partial credit
OpenSSF Scorecard: Pinned-Dependencies	10	Scorecard's 0–10 result, scaled to 10 points

Pinned-Dependencies is shared evidence: it adds supply-chain reproducibility information to the agent verification loop while remaining a full Security component. It is excluded when Scorecard is unavailable or reports n/a.

Evidence, not only equipment

Every other component here reads the repository's files: a task runner exists, a lockfile exists. Each is a proxy for a loop nobody has watched run.

Commits authored by a coding agent — or credited to one in a Co-authored-by trailer — are the one available record that the loop was actually closed. A change was proposed, verified, and merged with an agent in the loop, and the project's own history says so.

A dependency bot's pull request is machine-authored too, and it must clear the same gates: the tests run, the checks pass, it merges. A project already absorbing that traffic has shown the pathway an agent needs, which is why automated maintenance is scored beside agent practice rather than folded into it — either can be present without the other.

What counts is the observed commits, not the configuration file. A dependabot.yml can sit in a repository with the integration switched off, and Renovate is commonly configured outside the repository altogether: two widely used projects run it across roughly a third of their commits while carrying no recognizable config file at all. Only bots that author their own content qualify — a robot that merges other people's work is not writing changes.

The same commits count against a repository in development activity, where automation crowding out human work is a warning sign. That is deliberate. The two metrics ask different questions: whether people still maintain a project, and whether machines can contribute to it. Both answers can be true at once.

This evidences adoption, not autonomy. A maintainer working interactively with an agent leaves the same trace as an unattended run, and public data cannot tell the two apart, which is why the component carries a modest weight and makes no claim about how much of the work the agent did. Detection also recognizes only the agents it knows about, so a project using an unrecognized tool simply reads as having no evidence — never as failing.

The component is excluded rather than scored as zero when no commit sample was collected.

Why the toolchain counts as a bootstrap

cargo test and go test ./... are the canonical verify loops of their languages. Crediting only task runners charged those ecosystems for having better defaults — well-run Rust projects scored zero on this component for shipping no Makefile. package.json and pyproject.toml are deliberately not credited: npm and Python define no universal test command, and whether a given project defines one lives inside file contents this metric does not read.

Why this loop decides agent usefulness

Each component removes one failure mode of autonomous work:

Bootstrap — the agent can go from clone to running without archaeology.
Tests — the agent can prove a change did what was intended.
Lint and types — entire classes of error are caught mechanically, before review.
Reproducible environment — "works in the agent's sandbox" means works elsewhere too.

Notably, this is the same infrastructure that serves human contributors — the metric rewards no agent-specific exotica beyond what a disciplined project already has. A strong Engineering Quality project typically starts strong here.

Reading the result

Signals confirm the loop's existence, not its speed or coverage — the presence-based honesty rule of the whole badge.
A high verify loop with a stub agent context usually means a well-engineered project that simply has not written agent guidance yet — the cheapest possible improvement.

Improving the value

Add a Makefile (or justfile/Taskfile) exposing install, test, and lint targets.
Keep an automated test suite runnable locally in one command.
Adopt type checking — even a minimal mypy/tsconfig counts.
Commit a lockfile, Dockerfile, or devcontainer for environment reproducibility.
Where agents are already used, keep the Co-authored-by trailer the tools add by default. Stripping it removes the only public evidence that the loop closes in practice.