crates.io
spider-rs/spider
Foundational low latency web data collecting in Rust
MITOct 20, 2026 · metrics 2.10.0
All tags
Catalogue tag
#crawler
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
114 records
PyPI
z-mio/ParseHub
轻量、异步、开箱即用的社交媒体聚合解析库
MITOct 20, 2026 · metrics 2.10.0
npm
microlinkhq/top-user-agents
Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.
MITSep 16, 2026 · metrics 2.10.0
npm
squirrelscan/squirrelscan
The website QA tool for your coding agent. 295+ audit rules across SEO, performance, security, accessibility and agent experience.
MITSep 19, 2026 · metrics 2.10.0
Packagist
JayBizzle/Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
MITAug 27, 2026 · metrics 2.10.0
npm
NeuraLegion/bright-cli
Command Line Interface (CLI) tool for BrightSec's solutions.
MITSep 11, 2026 · metrics 2.10.0
PyPI
Liu233w/ojhunt-lite
A lightweight async Python tool for querying Online Judge (OJ) statistics across multiple platforms. Track your accepted problems (AC) and total submissions from 29+ competitive programming platforms.
BSD-2-ClauseSep 14, 2026 · metrics 2.10.0
npm
MrAdex77/google-play-scraper
Google Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.
MITSep 13, 2026 · metrics 2.10.0
npm
apmantza/pi-webaio
AiO webtools for Pi: Search, Fetch, Pull | no API keys
MITSep 8, 2026 · metrics 2.10.0
PyPI
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
MITSep 12, 2026 · metrics 2.10.0
Packagist
spatie/crawler
https://spatie.be/docs/crawler
MITSep 12, 2026 · metrics 2.10.0
npm
51Degrees/device-detection-node
Device detection engines for Node.js implementation of the 51Degrees Pipeline API
Custom licenseSep 9, 2026 · metrics 2.10.0
Maven
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Apache-2.0Oct 21, 2026 · metrics 2.10.0
npm · crates.io
StopMakingThatBigFace/node-wreq
🎭 Typescript HTTP client with native TLS, HTTP2, JA3, JA4 browser impersonation backed by wreq's Rust core
MITOct 15, 2026 · metrics 2.10.0
PyPI
hect0x7/JMComic-Crawler-Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
MITSep 15, 2026 · metrics 2.10.0
PyPI · npm
kameleo-io/kameleo
Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.
No licenseOct 7, 2026 · metrics 2.10.0
NuGet
soenneker/soenneker.playwrights.crawler
A configurable Playwright crawler with rich stealth and control options.
MITOct 21, 2026 · metrics 2.10.0
NuGet
JaCraig/Spidey
A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Apache-2.0Oct 19, 2026 · metrics 2.10.0
PyPI
PyPtt/PyPtt
The best PTT library
LGPL-3.0Sep 11, 2026 · metrics 2.10.0
PyPI
adbar/courlan
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Apache-2.0Sep 12, 2026 · metrics 2.10.0
npm
firecrawl/cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
No licenseSep 13, 2026 · metrics 2.10.0
PyPI
JorG18/agentcrawl
Self-hosted web scraping and Markdown extraction for AI agents
Apache-2.0Oct 9, 2026 · metrics 2.10.0
npm
cosmocoder/mcp-web-docs
Self-hosted MCP server to index and search any documentation site—public or private
MITSep 13, 2026 · metrics 2.10.0
npm
iannuttall/seo
The only SEO skill your agent needs. 70+ SEO audit tools through a local CLI and MCP server, using your own crawl, Search Console, and GA4 data.
Apache-2.0Sep 15, 2026 · metrics 2.10.0
npm
just-every/mcp-read-website-fast
Quickly reads webpages and converts to markdown for fast, token efficient web scraping
MITSep 20, 2026 · metrics 2.10.0
npm
qualweb/qualweb
No repository description published.
ISCSep 5, 2026 · metrics 2.10.0
PyPI
rmax/scrapy-redis
Redis-based components for Scrapy.
MITSep 19, 2026 · metrics 2.10.0
npm
rodrigogs/pornhub
pornhub API library
BSD-3-ClauseSep 18, 2026 · metrics 2.10.0
Go
gosom/scrapemate
Golang Crawling and scraping framework
MITSep 15, 2026 · metrics 2.10.0
PyPI
moskrc/crawlerdetect
🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
MITSep 14, 2026 · metrics 2.10.0