npm
NeuraLegion/bright-cli
Command Line Interface (CLI) tool for BrightSec's solutions.
MITJul 20, 2026 · metrics 2.10.0
All tags
Catalogue tag
#crawler
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
86 records
npm
MrAdex77/google-play-scraper
Google Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.
MITJul 23, 2026 · metrics 2.10.0
Maven
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Apache-2.0Sep 5, 2026 · metrics 2.10.0
PyPI
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
MITJul 20, 2026 · metrics 2.10.0
npm
51Degrees/device-detection-node
Device detection engines for Node.js implementation of the 51Degrees Pipeline API
Custom licenseJul 16, 2026 · metrics 2.10.0
NuGet
JaCraig/Spidey
A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Apache-2.0Aug 21, 2026 · metrics 2.10.0
Packagist
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
MITAug 4, 2026 · metrics 2.10.0
npm · crates.io
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
AGPL-3.0Jul 26, 2026 · metrics 2.10.0
NuGet
soenneker/soenneker.playwrights.crawler
A configurable Playwright crawler with rich stealth and control options.
MITAug 31, 2026 · metrics 2.10.0
PyPI
Liu233w/ojhunt-lite
A lightweight async Python tool for querying Online Judge (OJ) statistics across multiple platforms. Track your accepted problems (AC) and total submissions from 29+ competitive programming platforms.
BSD-2-ClauseJul 29, 2026 · metrics 2.10.0
PyPI
PyPtt/PyPtt
The best PTT library
LGPL-3.0Jul 19, 2026 · metrics 2.10.0
npm
firecrawl/cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
No licenseJul 26, 2026 · metrics 2.10.0
npm
iannuttall/seo
The only SEO skill your agent needs. 70+ SEO audit tools through a local CLI and MCP server, using your own crawl, Search Console, and GA4 data.
Apache-2.0Aug 2, 2026 · metrics 2.10.0
npm
qualweb/qualweb
No repository description published.
ISCSep 5, 2026 · metrics 2.10.0
Go
vincentkoc/crawlkit
Shared Go infrastructure for local-first crawler archives.
MITJul 15, 2026 · metrics 2.10.0
PyPI
adbar/courlan
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Apache-2.0Jul 21, 2026 · metrics 2.10.0
Maven
fanyong920/jvppeteer
Java API For Chrome and Firefox
Apache-2.0Jul 17, 2026 · metrics 2.10.0
Go
gosom/scrapemate
Golang Crawling and scraping framework
MITAug 2, 2026 · metrics 2.10.0
Go
krau/ManyACG
Collect, Download, Organize and Share your Favorite Anime Artworks.
AGPL-3.0Aug 22, 2026 · metrics 2.10.0
npm
cosmocoder/mcp-web-docs
Self-hosted MCP server to index and search any documentation site—public or private
MITJul 26, 2026 · metrics 2.10.0
npm
ODATANO/NIGHTGATE
ODATANO-NIGHTGATE connects SAP systems to the Midnight privacy chain via a CAP-based OData V4 API, enabling confidential on-chain data access and privacy-preserving transaction execution
Apache-2.0Jul 18, 2026 · metrics 2.10.0
npm
apmantza/pi-webaio
AiO webtools for Pi: Search, Fetch, Pull
MITJul 15, 2026 · metrics 2.10.0
RubyGems
loadkpi/crawler_detect
Ruby gem to detect bots and crawlers via the user agent
MITSep 6, 2026 · metrics 2.10.0
npm
mysleekdesigns/crawlforge-mcp
28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
MITSep 5, 2026 · metrics 2.10.0
npm · crates.io · Go +1
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
MITJul 16, 2026 · metrics 2.10.0
npm
thecodrr/fdir
⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s
MITAug 4, 2026 · metrics 2.10.0
Go
uinaf/lincrawl
Local-first Linear work-graph archive CLI
MITJul 21, 2026 · metrics 2.10.0
NuGet
zzzprojects/html-agility-pack
Html Agility Pack (HAP) is a free and open-source HTML parser written in C# to read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files.
MITAug 22, 2026 · metrics 2.10.0
PyPI
moskrc/crawlerdetect
🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
MITJul 29, 2026 · metrics 2.10.0
PyPI
bitextor/bitextor
Bitextor generates translation memories from multilingual websites
GPL-3.0Jul 21, 2026 · metrics 2.10.0