All tags
Catalogue tag

#web-crawler

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

10 records
Tagged “web-crawler”Ranked by health index
npm
84Goodhealth index
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 24.8KJul 19, 2026
Apache-2.0Jul 19, 2026 · metrics 1.13.0
—
81Goodhealth index
mendableai/firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
TypeScript · Python★ 151.8KJul 16, 2026
AGPL-3.0Jul 16, 2026 · metrics 1.13.0
—
80Goodhealth index
firecrawl/firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
TypeScript★ 149.9K↓ 0/moJul 13, 2026
AGPL-3.0Jul 13, 2026 · metrics 1.13.0
npm
73Goodhealth index
KnockOutEZ/wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 3,201↓ 3,522/moJul 22, 2026
Custom licenseJul 22, 2026 · metrics 1.13.0
crates.io
71Goodhealth index
spider-rs/spider
Get web data for AI agents and LLMs
Rust★ 2,601↓ 52.8K/moJul 14, 2026
MITJul 14, 2026 · metrics 1.13.0
npm
68Moderatehealth index
microlinkhq/browserless
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
JavaScript★ 1,826Jul 18, 2026
MITJul 18, 2026 · metrics 1.13.0
crates.io · npm · Packagist +1
64Moderatehealth index
xberg-io/crawlberg
High-performance web crawling engine with bindings for 11 languages
Rust★ 140↓ 0/moJul 14, 2026
MITJul 14, 2026 · metrics 1.13.0
npm · crates.io · Go +1
58Moderatehealth index
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
Rust · Python · Go★ 26↓ 16.3K/moJul 16, 2026
MITJul 16, 2026 · metrics 1.13.0
npm
57Moderatehealth index
mysleekdesigns/crawlforge-mcp
26-tool MCP server for web scraping, crawling, deep research & autonomous extraction — clean Markdown & structured JSON for Claude, Cursor & any MCP client. 1,000 free credits, local-Ollama LLM support.
JavaScript★ 1↓ 2,975/moJul 14, 2026
MITJul 14, 2026 · metrics 1.13.0
Maven
25Criticalhealth index
ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Java★ 11.3KJul 21, 2026
MITJul 21, 2026 · metrics 1.13.0