#crawler

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

A CAP-based OData V4 gateway to the Midnight privacy chain, providing confidential on-chain data access and privacy-preserving transaction execution to any enterprise system

Ruby gem to detect bots and crawlers via the user agent

28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.

Local-first Linear work-graph archive CLI

Python, Javascript, and Rust libraries for the Spider Cloud API.

Enterprise-grade, multi-threaded SEO Crawler, Rule Engine, and Link Graph Analyzer. Built in TypeScript for speed, compliance, and deep site health audits.

Brazilian financial market data sources

Java API For Chrome and Firefox

nodejs SDK of the Olostep API

Pi extension for fast page scraping, recursive crawling, URL/site mapping, brand extraction, content diffing, PDF text extraction, and deterministic vertical extraction.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Automatic proxy pool for web scraping - crawls, validates, and rotates proxies with rate limiting and MITM support

Curated list of common web crawler and bot User-Agent strings used across the internet.

GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them

Jira issue graph CLI and Go toolkit — fetch, crawl, mirror, and serve Jira issues with their links, PRs, and relationships

Local-first failure diagnosis, auto collection, repair planning, AI handoff, and verification for Playwright, crawler, RPA, and agent workflows.

The set of tools for fast writing crawlers on the .NET.

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Convert a webpage into plaintext or markdown using the Chrome DevTool Protocol

Go CLI for extracting websites, repositories, sitemaps, and package docs into structured Markdown