Todas las etiquetas
Etiqueta del catálogo

#web-crawler

Todos los repositorios del registro público que llevan esta etiqueta, procedente de sus topics de GitHub o de las palabras clave que publican sus registros de paquetes. La salud se mide con la misma metodología versionada que el resto del registro.

29 registros
Con la etiqueta «web-crawler»Ordenado por índice de salud
npm
99Excepcionalíndice de salud
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 26.2K↓ 6.8M/mes19 oct 2026
Apache-2.019 oct 2026 · métricas 2.10.0
PyPI · npm
99Excepcionalíndice de salud
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python · MDX★ 9535↓ 975.8K/mes17 sept 2026
Apache-2.017 sept 2026 · métricas 2.10.0
PyPI
94Excepcionalíndice de salud
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ
Python★ 81.4K16 sept 2026
BSD-3-Clause16 sept 2026 · métricas 2.10.0
PyPI
94Excepcionalíndice de salud
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Python★ 31K16 sept 2026
MIT16 sept 2026 · métricas 2.10.0
Maven
94Excepcionalíndice de salud
apache/stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
Java★ 100819 oct 2026
Apache-2.019 oct 2026 · métricas 2.10.0
PyPI · Packagist · crates.io +2
94Excepcionalíndice de salud
firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
TypeScript · Python★ 181.1K↓ 5.8M/mes16 sept 2026
AGPL-3.016 sept 2026 · métricas 2.10.0
crates.io · npm · PyPI
91Excelenteíndice de salud
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1095↓ 10.6K/mes2 oct 2026
AGPL-3.02 oct 2026 · métricas 2.10.0
crates.io · npm · PyPI
91Excelenteíndice de salud
us/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1032↓ 10.7K/mes16 sept 2026
AGPL-3.016 sept 2026 · métricas 2.10.0
npm
89Excelenteíndice de salud
JustinBeckwith/linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
TypeScript★ 1269↓ 691.2K/mes3 oct 2026
MIT3 oct 2026 · métricas 2.10.0
npm
89Excelenteíndice de salud
firecrawl/firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
TypeScript · JavaScript★ 7332↓ 492.7K/mes28 ago 2026
MIT28 ago 2026 · métricas 2.10.0
npm · PyPI
89Excelenteíndice de salud
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
TypeScript★ 111724 ago 2026
AGPL-3.024 ago 2026 · métricas 2.10.0
npm · crates.io
87Excelenteíndice de salud
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Rust★ 2343↓ 1301/mes13 sept 2026
AGPL-3.013 sept 2026 · métricas 2.10.0
npm
84Excelenteíndice de salud
KnockOutEZ/wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 5237↓ 3896/mes12 sept 2026
Licencia propia12 sept 2026 · métricas 2.10.0
crates.io
84Excelenteíndice de salud
spider-rs/spider
Foundational low latency web data collecting in Rust
Rust★ 2790↓ 24.2K/mes20 oct 2026
MIT20 oct 2026 · métricas 2.10.0
PyPI
84Excelenteíndice de salud
tobocop2/lilbee
The whole local AI stack in one executable: it runs and manages local AI models across every GPU, and it's a search engine you can talk to, with cited answers from your files, code, and the web. MCP server for coding agents, web crawler, TUI, CLI, REST API, Python library. No Ollama or LM Studio needed, works with both.
Python★ 54↓ 3110/mes18 sept 2026
MIT18 sept 2026 · métricas 2.10.0
PyPI · npm
83Excelenteíndice de salud
ScrapeGraphAI/scrapegraph-py
Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction
Jupyter Notebook★ 86↓ 212.9K/mes7 sept 2026
MIT7 sept 2026 · métricas 2.10.0
Go
80Excelenteíndice de salud
kinorai/omnifeed
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
Go · HTML★ 1021 oct 2026
MIT21 oct 2026 · métricas 2.10.0
npm · PyPI · crates.io
78Buenoíndice de salud
Goldziher/basemind
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
Rust★ 99↓ 2971/mes14 sept 2026
MIT14 sept 2026 · métricas 2.10.0
npm
78Buenoíndice de salud
scrapfly/typescript-scrapfly
Official TypeScript/JavaScript SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Ships to npm, JSR, and Deno.
TypeScript★ 2317 oct 2026
Licencia propia17 oct 2026 · métricas 2.10.0
npm · Packagist · crates.io +1
78Buenoíndice de salud
xberg-io/crawlberg
High-performance web crawling engine with bindings for 11 languages
Rust★ 173↓ 1094/mes15 sept 2026
MIT15 sept 2026 · métricas 2.10.0
Maven
77Buenoíndice de salud
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Java★ 20521 oct 2026
Apache-2.021 oct 2026 · métricas 2.10.0
PyPI
73Buenoíndice de salud
JorG18/agentcrawl
Self-hosted web scraping and Markdown extraction for AI agents
Python★ 10↓ 2758/mes9 oct 2026
Apache-2.09 oct 2026 · métricas 2.10.0
Go
73Buenoíndice de salud
jonhadfield/ip-fetcher
Go library and CLI to fetch public IP ranges from cloud platforms, CDNs, crawler bots, monitoring probes and threat intelligence feeds
Go★ 412 sept 2026
MIT12 sept 2026 · métricas 2.10.0
Go
69Buenoíndice de salud
gosom/scrapemate
Golang Crawling and scraping framework
Go★ 21015 sept 2026
MIT15 sept 2026 · métricas 2.10.0
npm
69Buenoíndice de salud
mysleekdesigns/crawlforge-mcp
31 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
JavaScript★ 2↓ 5464/mes21 oct 2026
MIT21 oct 2026 · métricas 2.10.0
PyPI · npm · crates.io +1
62Moderadoíndice de salud
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
Rust · Python · Go★ 26↓ 365K/mes9 sept 2026
MIT9 sept 2026 · métricas 2.10.0
Go · RubyGems
60Moderadoíndice de salud
1mb-dev/lobster
Intelligent web stress testing with auto URL discovery. Crawls your app, validates performance under load, generates actionable reports. Zero config required.
Go★ 219 sept 2026
MIT19 sept 2026 · métricas 2.10.0
npm
38Débilíndice de salud
Aegis-Runner/AegisRunner
AegisRunner crawls any website from a single URL, discovers every page, form, and interaction, then uses AI to generate a complete Playwright test suite. Also runs accessibility, SEO, security, and performance audits on every page — no recording, no scripting, no setup.
JavaScript★ 1↓ 2534/mes13 sept 2026
MIT13 sept 2026 · métricas 2.10.0
Maven
23En riesgoíndice de salud
ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Java★ 11.4K27 ago 2026
MIT27 ago 2026 · métricas 2.10.0