All tags
Catalogue tag

#crawler

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

114 records
Tagged “crawler”Ranked by health index
npm
99Exceptionalhealth index
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript · MDX★ 26.2K↓ 6.8M/moOct 19, 2026
Apache-2.0Oct 19, 2026 · metrics 2.10.0
PyPI · npm
99Exceptionalhealth index
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python · MDX★ 9,535↓ 975.8K/moSep 17, 2026
Apache-2.0Sep 17, 2026 · metrics 2.10.0
PyPI · npm
97Exceptionalhealth index
apify/apify-sdk-python
Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.
Python · MDX★ 176↓ 232.6K/moSep 11, 2026
Apache-2.0Sep 11, 2026 · metrics 2.10.0
PyPI
94Exceptionalhealth index
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ
Python★ 81.4KSep 16, 2026
BSD-3-ClauseSep 16, 2026 · metrics 2.10.0
PyPI
94Exceptionalhealth index
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Python★ 31KSep 16, 2026
MITSep 16, 2026 · metrics 2.10.0
Maven
94Exceptionalhealth index
apache/stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
Java★ 1,008Oct 19, 2026
Apache-2.0Oct 19, 2026 · metrics 2.10.0
npm
94Exceptionalhealth index
apify/apify-client-js
Apify API client for JavaScript / Node.js.
TypeScript★ 90↓ 1.4M/moSep 20, 2026
Apache-2.0Sep 20, 2026 · metrics 2.10.0
PyPI · Packagist · crates.io +2
94Exceptionalhealth index
firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
TypeScript · Python★ 181.1K↓ 5.8M/moSep 16, 2026
AGPL-3.0Sep 16, 2026 · metrics 2.10.0
Go
94Exceptionalhealth index
projectdiscovery/katana
A next-generation crawling and spidering framework.
Go★ 17.7KOct 19, 2026
MITOct 19, 2026 · metrics 2.10.0
npm
92Excellenthealth index
apify/apify-sdk-js
Apify SDK monorepo
MDX · TypeScript★ 186↓ 178.2K/moSep 12, 2026
Apache-2.0Sep 12, 2026 · metrics 2.10.0
NuGet
92Excellenthealth index
hardkoded/puppeteer-sharp
Headless Chrome .NET API
C#★ 3,916Aug 28, 2026
MITAug 28, 2026 · metrics 2.10.0
PyPI
92Excellenthealth index
mikf/gallery-dl
Command-line program to download image galleries and collections from several image hosting sites
Python★ 19.7K↓ 320.4K/moSep 16, 2026
GPL-2.0Sep 16, 2026 · metrics 2.10.0
PyPI
91Excellenthealth index
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Python★ 6,718↓ 14M/moAug 28, 2026
Apache-2.0Aug 28, 2026 · metrics 2.10.0
crates.io · npm · PyPI
91Excellenthealth index
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1,095↓ 10.6K/moOct 2, 2026
AGPL-3.0Oct 2, 2026 · metrics 2.10.0
crates.io · npm · PyPI
91Excellenthealth index
us/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Rust★ 1,032↓ 10.7K/moSep 16, 2026
AGPL-3.0Sep 16, 2026 · metrics 2.10.0
PyPI · npm
90Excellenthealth index
apify/fingerprint-suite
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
TypeScript · JavaScript★ 2,599↓ 2M/moSep 17, 2026
Apache-2.0Sep 17, 2026 · metrics 2.10.0
crates.io
89Excellenthealth index
0x676e67/wreq
An ergonomic, privacy-aware Rust HTTP Client
Rust★ 1,076↓ 222.8K/moOct 20, 2026
Apache-2.0Oct 20, 2026 · metrics 2.10.0
npm
89Excellenthealth index
JustinBeckwith/linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
TypeScript★ 1,269↓ 691.2K/moOct 3, 2026
MITOct 3, 2026 · metrics 2.10.0
npm · crates.io
89Excellenthealth index
sqdshguy/wreq-js
HTTP client for Node.js with browser TLS fingerprint impersonation
TypeScript · Rust★ 402↓ 498.7K/moSep 23, 2026
MITSep 23, 2026 · metrics 2.10.0
Packagist
89Excellenthealth index
tomasnorre/crawler
Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.
PHP★ 58↓ 10.7K/moOct 20, 2026
GPL-3.0Oct 20, 2026 · metrics 2.10.0
npm · PyPI
89Excellenthealth index
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
TypeScript★ 1,117Aug 24, 2026
AGPL-3.0Aug 24, 2026 · metrics 2.10.0
Maven
88Excellenthealth index
TeamNewPipe/NewPipeExtractor
NewPipe's core library for extracting data from streaming sites
Java★ 1,972Sep 17, 2026
GPL-3.0Sep 17, 2026 · metrics 2.10.0
npm
88Excellenthealth index
zorillajs/zorilla
Zorilla is a modular plugin framework that extends Puppeteer and Playwright with additional functionality through a clean plugin architecture. Build powerful browser automation with composable plugins for stealth mode, captcha solving, ad blocking, and much more.
TypeScript · JavaScript★ 30↓ 25.5K/moSep 14, 2026
MITSep 14, 2026 · metrics 2.10.0
PyPI · crates.io
87Excellenthealth index
0x676e67/wreq-python
An ergonomic, privacy-aware Python HTTP Client
Rust · Python★ 1,441Sep 17, 2026
Apache-2.0Sep 17, 2026 · metrics 2.10.0
PyPI
87Excellenthealth index
DedSecInside/TorBot
Dark Web OSINT Tool
Python★ 4,726Aug 28, 2026
Custom licenseAug 28, 2026 · metrics 2.10.0
Packagist · npm
87Excellenthealth index
eliashaeussler/cache-warmup
🔥 PHP library to warm up caches of URLs located in XML sitemaps
PHP★ 77↓ 16.4K/moSep 9, 2026
GPL-3.0Sep 9, 2026 · metrics 2.10.0
npm
87Excellenthealth index
getmaxun/maxun
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
TypeScript★ 17.5KSep 16, 2026
AGPL-3.0Sep 16, 2026 · metrics 2.10.0
NuGet
86Excellenthealth index
51Degrees/device-detection-dotnet
Device detection services for 51Degrees Pipeline
C# · C++★ 11Oct 20, 2026
Custom licenseOct 20, 2026 · metrics 2.10.0
Go
86Excellenthealth index
openclaw/crawlkit
Shared Go infrastructure for local-first crawler archives.
Go · Python★ 59Sep 8, 2026
MITSep 8, 2026 · metrics 2.10.0
npm
84Excellenthealth index
KnockOutEZ/wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TypeScript★ 5,237↓ 3,896/moSep 12, 2026
Custom licenseSep 12, 2026 · metrics 2.10.0