Todas las etiquetas
Etiqueta del catálogo

#web-crawler

Todos los repositorios del registro público que llevan esta etiqueta, procedente de sus topics de GitHub o de las palabras clave que publican sus registros de paquetes. La salud se mide con la misma metodología versionada que el resto del registro.

21 registros
Con la etiqueta «web-crawler»Ordenado por índice de salud
npm
98Excepcionalíndice de salud
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 25.2K↓ 6M/mes5 ago 2026
Apache-2.05 ago 2026 · métricas 2.10.0
PyPI · npm
98Excepcionalíndice de salud
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python · MDX★ 942612 ago 2026
Apache-2.012 ago 2026 · métricas 2.10.0
PyPI
94Excepcionalíndice de salud
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Python★ 72.6K4 ago 2026
BSD-3-Clause4 ago 2026 · métricas 2.10.0
PyPI
94Excepcionalíndice de salud
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Python★ 29K5 ago 2026
MIT5 ago 2026 · métricas 2.10.0
Packagist · Hex · crates.io +2
94Excepcionalíndice de salud
firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
TypeScript · Python★ 161K↓ 3278/mes4 ago 2026
AGPL-3.04 ago 2026 · métricas 2.10.0
npm
89Excelenteíndice de salud
firecrawl/firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
TypeScript · JavaScript★ 7332↓ 492.7K/mes28 ago 2026
MIT28 ago 2026 · métricas 2.10.0
npm · PyPI
89Excelenteíndice de salud
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
TypeScript★ 111724 ago 2026
AGPL-3.024 ago 2026 · métricas 2.10.0
npm
87Excelenteíndice de salud
KnockOutEZ/wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 3201↓ 3522/mes22 jul 2026
Licencia propia22 jul 2026 · métricas 2.10.0
crates.io
87Excelenteíndice de salud
spider-rs/spider
Get web data for AI agents and LLMs
Rust★ 2669↓ 42K/mes22 ago 2026
MIT22 ago 2026 · métricas 2.10.0
npm · PyPI · crates.io
87Excelenteíndice de salud
us/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 518↓ 9070/mes3 ago 2026
AGPL-3.03 ago 2026 · métricas 2.10.0
npm
80Excelenteíndice de salud
microlinkhq/browserless
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
JavaScript★ 182618 jul 2026
MIT18 jul 2026 · métricas 2.10.0
Maven
78Buenoíndice de salud
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Java★ 2045 sept 2026
Apache-2.05 sept 2026 · métricas 2.10.0
Go
78Buenoíndice de salud
kinorai/omnifeed
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
Go★ 63 sept 2026
MIT3 sept 2026 · métricas 2.10.0
npm · crates.io · Packagist +1
78Buenoíndice de salud
xberg-io/crawlberg
High-performance web crawling engine with bindings for 11 languages
Rust★ 149↓ 685/mes1 ago 2026
MIT1 ago 2026 · métricas 2.10.0
npm · crates.io
75Buenoíndice de salud
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Rust★ 2066↓ 424/mes26 jul 2026
AGPL-3.026 jul 2026 · métricas 2.10.0
npm · PyPI · crates.io
75Buenoíndice de salud
Goldziher/basemind
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
Rust★ 69↓ 13.6K/mes27 jul 2026
MIT27 jul 2026 · métricas 2.10.0
Go
67Buenoíndice de salud
gosom/scrapemate
Golang Crawling and scraping framework
Go★ 2072 ago 2026
MIT2 ago 2026 · métricas 2.10.0
npm
63Moderadoíndice de salud
mysleekdesigns/crawlforge-mcp
28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
JavaScript★ 2↓ 5179/mes5 sept 2026
MIT5 sept 2026 · métricas 2.10.0
npm · crates.io · Go +1
63Moderadoíndice de salud
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
Rust · Python · Go★ 26↓ 16.3K/mes16 jul 2026
MIT16 jul 2026 · métricas 2.10.0
npm
41Débilíndice de salud
Aegis-Runner/AegisRunner
AegisRunner crawls any website from a single URL, discovers every page, form, and interaction, then uses AI to generate a complete Playwright test suite. Also runs accessibility, SEO, security, and performance audits on every page — no recording, no scripting, no setup.
JavaScript★ 0↓ 8920/mes25 jul 2026
MIT25 jul 2026 · métricas 2.10.0
Maven
23En riesgoíndice de salud
ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Java★ 11.4K27 ago 2026
MIT27 ago 2026 · métricas 2.10.0