crates.io
spider-rs/spider
Foundational low latency web data collecting in Rust
MIT20 oct 2026 · métricas 2.10.0
Todas las etiquetas
Etiqueta del catálogo
#crawler
Todos los repositorios del registro público que llevan esta etiqueta, procedente de sus topics de GitHub o de las palabras clave que publican sus registros de paquetes. La salud se mide con la misma metodología versionada que el resto del registro.
114 registros
PyPI
z-mio/ParseHub
轻量、异步、开箱即用的社交媒体聚合解析库
MIT20 oct 2026 · métricas 2.10.0
npm
microlinkhq/top-user-agents
Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.
MIT16 sept 2026 · métricas 2.10.0
npm
squirrelscan/squirrelscan
The website QA tool for your coding agent. 295+ audit rules across SEO, performance, security, accessibility and agent experience.
MIT19 sept 2026 · métricas 2.10.0
Packagist
JayBizzle/Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
MIT27 ago 2026 · métricas 2.10.0
npm
NeuraLegion/bright-cli
Command Line Interface (CLI) tool for BrightSec's solutions.
MIT11 sept 2026 · métricas 2.10.0
PyPI
Liu233w/ojhunt-lite
A lightweight async Python tool for querying Online Judge (OJ) statistics across multiple platforms. Track your accepted problems (AC) and total submissions from 29+ competitive programming platforms.
BSD-2-Clause14 sept 2026 · métricas 2.10.0
npm
MrAdex77/google-play-scraper
Google Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.
MIT13 sept 2026 · métricas 2.10.0
npm
apmantza/pi-webaio
AiO webtools for Pi: Search, Fetch, Pull | no API keys
MIT8 sept 2026 · métricas 2.10.0
PyPI
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
MIT12 sept 2026 · métricas 2.10.0
Packagist
spatie/crawler
https://spatie.be/docs/crawler
MIT12 sept 2026 · métricas 2.10.0
npm
51Degrees/device-detection-node
Device detection engines for Node.js implementation of the 51Degrees Pipeline API
Licencia propia9 sept 2026 · métricas 2.10.0
Maven
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Apache-2.021 oct 2026 · métricas 2.10.0
npm · crates.io
StopMakingThatBigFace/node-wreq
🎭 Typescript HTTP client with native TLS, HTTP2, JA3, JA4 browser impersonation backed by wreq's Rust core
MIT15 oct 2026 · métricas 2.10.0
PyPI
hect0x7/JMComic-Crawler-Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
MIT15 sept 2026 · métricas 2.10.0
PyPI · npm
kameleo-io/kameleo
Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.
Sin licencia7 oct 2026 · métricas 2.10.0
NuGet
soenneker/soenneker.playwrights.crawler
A configurable Playwright crawler with rich stealth and control options.
MIT21 oct 2026 · métricas 2.10.0
NuGet
JaCraig/Spidey
A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Apache-2.019 oct 2026 · métricas 2.10.0
PyPI
PyPtt/PyPtt
The best PTT library
LGPL-3.011 sept 2026 · métricas 2.10.0
PyPI
adbar/courlan
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Apache-2.012 sept 2026 · métricas 2.10.0
npm
firecrawl/cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
Sin licencia13 sept 2026 · métricas 2.10.0
PyPI
JorG18/agentcrawl
Self-hosted web scraping and Markdown extraction for AI agents
Apache-2.09 oct 2026 · métricas 2.10.0
npm
cosmocoder/mcp-web-docs
Self-hosted MCP server to index and search any documentation site—public or private
MIT13 sept 2026 · métricas 2.10.0
npm
iannuttall/seo
The only SEO skill your agent needs. 70+ SEO audit tools through a local CLI and MCP server, using your own crawl, Search Console, and GA4 data.
Apache-2.015 sept 2026 · métricas 2.10.0
npm
qualweb/qualweb
El repositorio no publica descripción.
ISC22 oct 2026 · métricas 2.10.0
npm
just-every/mcp-read-website-fast
Quickly reads webpages and converts to markdown for fast, token efficient web scraping
MIT20 sept 2026 · métricas 2.10.0
PyPI
rmax/scrapy-redis
Redis-based components for Scrapy.
MIT19 sept 2026 · métricas 2.10.0
npm
rodrigogs/pornhub
pornhub API library
BSD-3-Clause18 sept 2026 · métricas 2.10.0
Go
gosom/scrapemate
Golang Crawling and scraping framework
MIT15 sept 2026 · métricas 2.10.0
PyPI
moskrc/crawlerdetect
🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
MIT14 sept 2026 · métricas 2.10.0