All tags
Catalogue tag

#crawl

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

11 records
Tagged “crawl”Ranked by health index
npm · PyPI · crates.io
87Excellenthealth index
us/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 518↓ 9,070/moAug 3, 2026
AGPL-3.0Aug 3, 2026 · metrics 2.10.0
PyPI
78Goodhealth index
tavily-ai/tavily-python
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
Python★ 1,375↓ 8.5M/moAug 27, 2026
MITAug 27, 2026 · metrics 2.10.0
PyPI
75Goodhealth index
ScrapingBee/scrapingbee-cli
No repository description published.
Python★ 107Aug 2, 2026
MITAug 2, 2026 · metrics 2.10.0
npm
67Goodhealth index
superagents-lab/search1api-mcp
Official Search1API MCP server for web search, news, crawling, sitemaps, and trends—hosted with OAuth 2.1 or local via npm.
TypeScript · JavaScript★ 173↓ 2,722/moAug 24, 2026
MITAug 24, 2026 · metrics 2.10.0
npm
63Moderatehealth index
mysleekdesigns/crawlforge-mcp
28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
JavaScript★ 2↓ 5,179/moSep 5, 2026
MITSep 5, 2026 · metrics 2.10.0
npm
63Moderatehealth index
tavily-ai/tavily-n8n-node
A node for n8n that integrates the Tavily API, enabling powerful web search and content extraction within your no-code automation workflows.
TypeScript★ 20↓ 82.8K/moSep 4, 2026
MITSep 4, 2026 · metrics 2.10.0
npm
60Moderatehealth index
tavily-ai/ai-sdk
AI SDK tools for Tavily, built for Vercel's AI SDK v5.
TypeScript★ 5↓ 69.5K/moSep 4, 2026
MITSep 4, 2026 · metrics 2.10.0
npm
59Moderatehealth index
brandonkramer/pi-scraper
Pi extension for fast page scraping, recursive crawling, URL/site mapping, brand extraction, content diffing, PDF text extraction, and deterministic vertical extraction.
TypeScript · HTML★ 7↓ 607/moAug 28, 2026
MITAug 28, 2026 · metrics 2.10.0
Go
51Moderatehealth index
0xMassi/webclaw-go
Official Go SDK for the Webclaw web extraction API
Go★ 1Jul 24, 2026
MITJul 24, 2026 · metrics 2.10.0
npm
31At Riskhealth index
ddoojoang/ttj-skills-playwright
No repository description published.
TypeScript · JavaScript★ 1↓ 3,696/moJul 29, 2026
No licenseJul 29, 2026 · metrics 2.10.0