Packagist
mvdbos/php-spider
A configurable and extensible PHP web spider
MITOct 19, 2026 · metrics 2.10.0
All tags
Catalogue tag
#crawler
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
114 records
npm
mysleekdesigns/crawlforge-mcp
31 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
MITOct 21, 2026 · metrics 2.10.0
Packagist
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
MITSep 16, 2026 · metrics 2.10.0
PyPI
scrapy-plugins/scrapy-zyte-smartproxy
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
BSD-3-ClauseSep 20, 2026 · metrics 2.10.0
Go
x-way/crawlerdetect
Golang module to detect bots and crawlers via the user agent
MITSep 9, 2026 · metrics 2.10.0
Go
krau/ManyACG
Collect, Download, Organize and Share your Favorite Anime Artworks.
AGPL-3.0Oct 20, 2026 · metrics 2.10.0
NuGet
zzzprojects/html-agility-pack
Html Agility Pack (HAP) is a free and open-source HTML parser written in C# to read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files.
MITOct 20, 2026 · metrics 2.10.0
Go · npm
Autumn-27/ScopeSentry
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
No licenseSep 18, 2026 · metrics 2.10.0
npm
ODATANO/NIGHTGATE
A CAP-based OData V4 gateway to the Midnight privacy chain, providing confidential on-chain data access and privacy-preserving transaction execution to any enterprise system
Apache-2.0Sep 10, 2026 · metrics 2.10.0
RubyGems
loadkpi/crawler_detect
Ruby gem to detect bots and crawlers via the user agent
MITSep 6, 2026 · metrics 2.10.0
Go
uinaf/lincrawl
Local-first Linear work-graph archive CLI
MITJul 21, 2026 · metrics 2.10.0
PyPI · npm · crates.io +1
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
MITSep 9, 2026 · metrics 2.10.0
npm
codepurse/SEOCORE
Enterprise-grade, multi-threaded SEO Crawler, Rule Engine, and Link Graph Analyzer. Built in TypeScript for speed, compliance, and deep site health audits.
MITSep 9, 2026 · metrics 2.10.0
PyPI
gabrielguarisa/brdata
Brazilian financial market data sources
No licenseSep 19, 2026 · metrics 2.10.0
Maven
fanyong920/jvppeteer
Java API For Chrome and Firefox
Apache-2.0Sep 10, 2026 · metrics 2.10.0
npm
olostep-api/olostep-js
nodejs SDK of the Olostep API
No licenseSep 15, 2026 · metrics 2.10.0
npm
brandonkramer/pi-scraper
Pi extension for fast page scraping, recursive crawling, URL/site mapping, brand extraction, content diffing, PDF text extraction, and deterministic vertical extraction.
MITOct 20, 2026 · metrics 2.10.0
PyPI
codelucas/newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
MITSep 17, 2026 · metrics 2.10.0
Packagist
JayBizzle/Laravel-Crawler-Detect
A Laravel wrapper for CrawlerDetect - the web crawler detection library
MITSep 14, 2026 · metrics 2.10.0
Go
naiba/proxy-in-a-box
Automatic proxy pool for web scraping - crawls, validates, and rotates proxies with rate limiting and MITM support
MITOct 19, 2026 · metrics 2.10.0
npm
Kikobeats/top-crawler-agents
Curated list of common web crawler and bot User-Agent strings used across the internet.
MITSep 19, 2026 · metrics 2.10.0
Maven · npm
seart-group/ghs
GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them
MITSep 19, 2026 · metrics 2.10.0
Go
neumachen/gojira
Jira issue graph CLI and Go toolkit — fetch, crawl, mirror, and serve Jira issues with their links, PRs, and relationships
MITSep 19, 2026 · metrics 2.10.0
PyPI
tobybgy-lsd/web-agent-runtime-bench
Local-first failure diagnosis, auto collection, repair planning, AI handoff, and verification for Playwright, crawler, RPA, and agent workflows.
Custom licenseSep 13, 2026 · metrics 2.10.0
NuGet
Laraue/Laraue.Crawling
The set of tools for fast writing crawlers on the .NET.
MITOct 4, 2026 · metrics 2.10.0
Packagist
jae-jae/QueryList
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
No licenseSep 15, 2026 · metrics 2.10.0
Go
mjc-gh/virgo
Convert a webpage into plaintext or markdown using the Chrome DevTool Protocol
BSD-3-ClauseSep 10, 2026 · metrics 2.10.0
Go
quantmind-br/repodocs
Go CLI for extracting websites, repositories, sitemaps, and package docs into structured Markdown
MITAug 25, 2026 · metrics 2.10.0
Packagist · npm
duzun/hQuery.php
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
MITSep 14, 2026 · metrics 2.10.0
npm
webscraping-ai/webscraping-ai-mcp-server
A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.
No licenseSep 12, 2026 · metrics 2.10.0