All tags
Catalogue tag

#robots-txt

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

7 records
Tagged “robots-txt”Ranked by health index
npm
91Excellenthealth index
nuxt-modules/robots
Tame the robots crawling and indexing your Nuxt site.
TypeScript★ 520↓ 763.2K/moJul 25, 2026
MITJul 25, 2026 · metrics 2.10.0
PyPI
78Goodhealth index
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
DIGITAL Command Language★ 91↓ 22.3M/moAug 8, 2026
BSD-3-ClauseAug 8, 2026 · metrics 2.10.0
Packagist
77Goodhealth index
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
PHP★ 258↓ 785.7K/moAug 4, 2026
MITAug 4, 2026 · metrics 2.10.0
npm
75Goodhealth index
CyberCraftBD/power-seo
Power-SEO: AI-powered, enterprise-grade SEO toolkit for React - A modern & best alternative to Yoast, RankMath & AIOSEO for SPA and SSR applications.
TypeScript★ 11↓ 10.7K/moAug 20, 2026
MITAug 20, 2026 · metrics 2.10.0
PyPI
73Goodhealth index
pip-install-python/dash-improve-my-llms
Automate page creation for an llms.txt, page.json and architecture.txt. Allow llms the ability to read and understand your dash applications pages.
Python★ 8↓ 2,907/moAug 20, 2026
MITAug 20, 2026 · metrics 2.10.0
Go
38Weakhealth index
11lunaric11/securitychecker
Recon tool for pentesters & bug bounty hunters: check robots.txt, security.txt (RFC 9116) and /.well-known/ across many targets. CLI + web UI, single static Go binary.
Go · HTML★ 2Jul 18, 2026
MITJul 18, 2026 · metrics 2.10.0
Packagist
30At Riskhealth index
mguinea/laravel-robots
Laravel package to manage robots
PHP★ 15↓ 1,302/moAug 13, 2026
MITAug 13, 2026 · metrics 2.10.0