All tags
Catalogue tag

#crawler

Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.

42 records
Tagged “crawler”Ranked by health index
npm
54Moderatehealth index
codepurse/SEOCORE
Enterprise-grade, multi-threaded SEO Crawler, Rule Engine, and Link Graph Analyzer. Built in TypeScript for speed, compliance, and deep site health audits.
TypeScript★ 109Jul 17, 2026
MITJul 17, 2026 · metrics 1.13.0
crates.io
51Moderatehealth index
spider-rs/spider_firewall
Firewall for Rust
Rust★ 1↓ 2,668/moJul 16, 2026
MITJul 16, 2026 · metrics 1.13.0
npm
51Moderatehealth index
webscraping-ai/webscraping-ai-mcp-server
A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.
JavaScript★ 44↓ 560/moJul 22, 2026
No licenseJul 22, 2026 · metrics 1.13.0
npm
49At riskhealth index
NeuraLegion/bright-cli
Command Line Interface (CLI) tool for BrightSec's solutions.
TypeScript★ 22↓ 3,745/moJul 20, 2026
MITJul 20, 2026 · metrics 1.13.0
Go
49At riskhealth index
mjc-gh/virgo
Convert a webpage into plaintext or markdown using the Chrome DevTool Protocol
Go · JavaScript★ 1Jul 17, 2026
BSD-3-ClauseJul 17, 2026 · metrics 1.13.0
Maven
48At riskhealth index
code4craft/webmagic
A scalable web crawler framework for Java.
Java · HTML★ 11.7KJul 20, 2026
Apache-2.0Jul 20, 2026 · metrics 1.13.0
Packagist
46At riskhealth index
crawlbase/crawlbase-php
A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API
PHP★ 16↓ 2,779/moJul 15, 2026
Apache-2.0Jul 15, 2026 · metrics 1.13.0
crates.io
44At riskhealth index
zekroTJA/r34-crawler
A simple CLI tool to fetch and download images from rule34.xxx
Rust★ 4Jul 22, 2026
MITJul 22, 2026 · metrics 1.13.0
npm
35At riskhealth index
crawlbase/crawlbase-node
Fast dependency free library for Crawlbase API
JavaScript★ 9Jul 19, 2026
Apache-2.0Jul 19, 2026 · metrics 1.13.0
Packagist
34At riskhealth index
JayBizzle/Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
PHP★ 2,374↓ 2.3M/moJul 20, 2026
MITJul 20, 2026 · metrics 1.13.0
PyPI
34At riskhealth index
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
Cython · Python★ 1,656Jul 20, 2026
MITJul 20, 2026 · metrics 1.13.0
Maven
25Criticalhealth index
ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Java★ 11.3KJul 21, 2026
MITJul 21, 2026 · metrics 1.13.0