All tags
Catalogue tag

#robots-txt

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

10 records
Tagged “robots-txt”Ranked by health index
npm
91Excellenthealth index
nuxt-modules/robots
Tame the robots crawling and indexing your Nuxt site.
TypeScript★ 526↓ 847.9K/moSep 13, 2026
MITSep 13, 2026 · metrics 2.10.0
PyPI
86Excellenthealth index
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
DIGITAL Command Language★ 95↓ 6.3M/moSep 17, 2026
BSD-3-ClauseSep 17, 2026 · metrics 2.10.0
npm
71Goodhealth index
CyberCraftBD/power-seo
Power-SEO: AI-powered, enterprise-grade SEO toolkit for React - A modern & best alternative to Yoast, RankMath & AIOSEO for SPA and SSR applications.
TypeScript★ 16↓ 5,250/moOct 19, 2026
MITOct 19, 2026 · metrics 2.10.0
PyPI
69Goodhealth index
pip-install-python/dash-improve-my-llms
Automate page creation for an llms.txt, page.json and architecture.txt. Allow llms the ability to read and understand your dash applications pages.
Python★ 8↓ 1,085/moOct 19, 2026
MITOct 19, 2026 · metrics 2.10.0
Packagist
69Goodhealth index
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
PHP★ 258↓ 945.6K/moSep 16, 2026
MITSep 16, 2026 · metrics 2.10.0
npm
60Moderatehealth index
aryasaatvik/pagegraph
Route-declared SEO graph and audit toolkit for TanStack Start: sitemap/robots, React head, JSON-LD, Vite coverage gate, live audit, and Jev-backed link decisions.
TypeScript★ 1↓ 3,892/moOct 5, 2026
MITOct 5, 2026 · metrics 2.10.0
NuGet
53Moderatehealth index
k3ldar/.NetCorePluginManager
.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications
C#★ 31Sep 15, 2026
GPL-3.0Sep 15, 2026 · metrics 2.10.0
Go
35Weakhealth index
11lunaric11/securitychecker
Recon tool for pentesters & bug bounty hunters: check robots.txt, security.txt (RFC 9116) and /.well-known/ across many targets. CLI + web UI, single static Go binary.
Go · HTML★ 2Sep 10, 2026
MITSep 10, 2026 · metrics 2.10.0
npm
30At Riskhealth index
trybyte-app/robotstxt-ts-port
Google's robots.txt parser ported to TypeScript - RFC 9309 compliant
TypeScript★ 2↓ 14.9K/moOct 11, 2026
Apache-2.0Oct 11, 2026 · metrics 2.10.0
Packagist
12Criticalhealth index
mguinea/laravel-robots
Laravel package to manage robots
PHP★ 15↓ 2,094/moSep 18, 2026
MITSep 18, 2026 · metrics 2.10.0