apify/crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 24.8KJul 19, 2026
PyPI · npm82Goodhealth index
apify/apify-sdk-pythonApify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.
Python · MDX★ 173Jul 20, 2026
d4vinci/scrapling🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Python★ 69.4K↓ 863.2K/moJul 14, 2026
mendableai/firecrawlThe API to search, scrape, and interact with the web at scale. 🔥
TypeScript · Python★ 151.8KJul 16, 2026
firecrawl/firecrawlThe API to search, scrape, and interact with the web at scale. 🔥
TypeScript★ 149.9K↓ 0/moJul 13, 2026
apify/apify-sdk-jsApify SDK monorepo
MDX · TypeScript★ 180↓ 149.3K/moJul 20, 2026
Packagist · npm76Goodhealth index
eliashaeussler/cache-warmup🔥 PHP library to warm up caches of URLs located in XML sitemaps
PHP★ 77↓ 14.1K/moJul 16, 2026
C#★ 3,900Jul 17, 2026
KnockOutEZ/wigoloThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 3,201↓ 3,522/moJul 22, 2026
Packagist73Goodhealth index
tomasnorre/crawlerLibraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.
PHP★ 57↓ 10.1K/moJul 13, 2026
adbar/trafilaturaPython & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Python★ 6,312Jul 18, 2026
Packagist71Goodhealth index
spatie/crawlerhttps://spatie.be/docs/crawler
PHP★ 2,829↓ 751.2K/moJul 20, 2026
crates.io71Goodhealth index
spider-rs/spiderGet web data for AI agents and LLMs
Rust★ 2,601↓ 52.8K/moJul 14, 2026
npm67Moderatehealth index
MrAdex77/google-play-scraperGoogle Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.
TypeScript★ 5↓ 4,236/moJul 23, 2026
npm66Moderatehealth index
51Degrees/device-detection-nodeDevice detection engines for Node.js implementation of the 51Degrees Pipeline API
C++ · JavaScript★ 2↓ 254.8K/moJul 16, 2026
PyPI65Moderatehealth index
z-mio/parsehub轻量、异步、开箱即用的社交媒体聚合解析库
Python★ 125↓ 7,961/moJul 13, 2026
PyPI64Moderatehealth index
PyPtt/PyPttThe best PTT library
Python★ 726↓ 2,592/moJul 19, 2026
PyPI63Moderatehealth index
adbar/courlanClean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Python★ 177Jul 21, 2026
Maven62Moderatehealth index
fanyong920/jvppeteerJava API For Chrome and Firefox
Java★ 808Jul 17, 2026
openclaw/crawlkitShared Go infrastructure for local-first crawler archives.
Go★ 51↓ 0/moJul 13, 2026
vincentkoc/crawlkitShared Go infrastructure for local-first crawler archives.
Go · Python★ 52Jul 15, 2026
npm60Moderatehealth index
thecodrr/fdir⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s
TypeScript★ 1,726↓ 612M/moJul 22, 2026
npm59Moderatehealth index
ODATANO/NIGHTGATEODATANO-NIGHTGATE connects SAP systems to the Midnight privacy chain via a CAP-based OData V4 API, enabling confidential on-chain data access and privacy-preserving transaction execution
TypeScript · JavaScript★ 3↓ 3,520/moJul 18, 2026
npm59Moderatehealth index
apmantza/pi-webaioAiO webtools for Pi: Search, Fetch, Pull
TypeScript · JavaScript★ 11↓ 1,369/moJul 15, 2026
krau/manyacgCollect, Download, Organize and Share your Favorite Anime Artworks.
Go★ 109↓ 0/moJul 13, 2026
npm · crates.io · Go +158Moderatehealth index
spider-rs/spider-clientsPython, Javascript, and Rust libraries for the Spider Cloud API.
Rust · Python · Go★ 26↓ 16.3K/moJul 16, 2026
npm57Moderatehealth index
mysleekdesigns/crawlforge-mcp26-tool MCP server for web scraping, crawling, deep research & autonomous extraction — clean Markdown & structured JSON for Claude, Cursor & any MCP client. 1,000 free credits, local-Ollama LLM support.
JavaScript★ 1↓ 2,975/moJul 14, 2026
uinaf/lincrawlLocal-first Linear work-graph archive CLI
Go★ 1Jul 21, 2026
PyPI55Moderatehealth index
bitextor/bitextorBitextor generates translation memories from multilingual websites
Python · Shell★ 299Jul 21, 2026
x-way/crawlerdetectGolang module to detect bots and crawlers via the user agent
Go★ 70Jul 17, 2026