npm99Exceptionalhealth index
apify/crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 26.2K↓ 6.8M/moOct 19, 2026
PyPI · npm99Exceptionalhealth index
apify/crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python · MDX★ 9,535↓ 975.8K/moSep 17, 2026
PyPI94Exceptionalhealth index
D4Vinci/Scrapling🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ
Python★ 81.4KSep 16, 2026
PyPI94Exceptionalhealth index
Python★ 31KSep 16, 2026
Maven94Exceptionalhealth index
Java★ 1,008Oct 19, 2026
PyPI · Packagist · crates.io +294Exceptionalhealth index
TypeScript · Python★ 181.1K↓ 5.8M/moSep 16, 2026
crates.io · npm · PyPI91Excellenthealth index
fastcrw/crwFast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1,095↓ 10.6K/moOct 2, 2026
crates.io · npm · PyPI91Excellenthealth index
us/crwFast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1,032↓ 10.7K/moSep 16, 2026
npm89Excellenthealth index
JustinBeckwith/linkinatorBroken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
TypeScript★ 1,269↓ 691.2K/moOct 3, 2026
npm89Excellenthealth index
TypeScript · JavaScript★ 7,332↓ 492.7K/moAug 28, 2026
npm · PyPI89Excellenthealth index
TypeScript★ 1,117Aug 24, 2026
npm · crates.io87Excellenthealth index
0xMassi/webclawFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Rust★ 2,343↓ 1,301/moSep 13, 2026
npm84Excellenthealth index
KnockOutEZ/wigoloThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 5,237↓ 3,896/moSep 12, 2026
crates.io84Excellenthealth index
Rust★ 2,790↓ 24.2K/moOct 20, 2026
PyPI84Excellenthealth index
tobocop2/lilbeeThe whole local AI stack in one executable: it runs and manages local AI models across every GPU, and it's a search engine you can talk to, with cited answers from your files, code, and the web. MCP server for coding agents, web crawler, TUI, CLI, REST API, Python library. No Ollama or LM Studio needed, works with both.
Python★ 54↓ 3,110/moSep 18, 2026
PyPI · npm83Excellenthealth index
ScrapeGraphAI/scrapegraph-pyOfficial Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction
Jupyter Notebook★ 86↓ 212.9K/moSep 7, 2026
Go80Excellenthealth index
kinorai/omnifeedLLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
Go · HTML★ 10Oct 21, 2026
npm · PyPI · crates.io78Goodhealth index
Goldziher/basemindFull AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
Rust★ 99↓ 2,971/moSep 14, 2026
Norconex/crawlerNorconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Java★ 204Sep 5, 2026
scrapfly/typescript-scrapflyOfficial TypeScript/JavaScript SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Ships to npm, JSR, and Deno.
TypeScript★ 23Oct 17, 2026
npm · Packagist · crates.io +178Goodhealth index
Rust★ 173↓ 1,094/moSep 15, 2026
Python★ 10↓ 2,758/moOct 9, 2026
jonhadfield/ip-fetcherGo library and CLI to fetch public IP ranges from cloud platforms, CDNs, crawler bots, monitoring probes and threat intelligence feeds
Go★ 4Sep 12, 2026
Go★ 210Sep 15, 2026
npm63Moderatehealth index
mysleekdesigns/crawlforge-mcp28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
JavaScript★ 2↓ 5,179/moSep 5, 2026
PyPI · npm · crates.io +162Moderatehealth index
Rust · Python · Go★ 26↓ 365K/moSep 9, 2026
Go · RubyGems60Moderatehealth index
1mb-dev/lobsterIntelligent web stress testing with auto URL discovery. Crawls your app, validates performance under load, generates actionable reports. Zero config required.
Go★ 2Sep 19, 2026
Aegis-Runner/AegisRunnerAegisRunner crawls any website from a single URL, discovers every page, form, and interaction, then uses AI to generate a complete Playwright test suite. Also runs accessibility, SEO, security, and performance audits on every page — no recording, no scripting, no setup.
JavaScript★ 1↓ 2,534/moSep 13, 2026
Java★ 11.4KAug 27, 2026