crates.io
spider-rs/spider
Foundational low latency web data collecting in Rust
MIT20. Okt. 2026 · Metriken 2.10.0
Alle Tags
Katalog-Tag
#crawler
Alle Repositories im öffentlichen Register, die dieses Tag tragen — aus ihren GitHub-Topics oder den von ihren Paket-Registries veröffentlichten Schlagwörtern. Die Gesundheit wird nach derselben versionierten Methodik gemessen wie im übrigen Register.
114 Einträge
PyPI
z-mio/ParseHub
轻量、异步、开箱即用的社交媒体聚合解析库
MIT20. Okt. 2026 · Metriken 2.10.0
npm
microlinkhq/top-user-agents
Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.
MIT16. Sept. 2026 · Metriken 2.10.0
npm
squirrelscan/squirrelscan
The website QA tool for your coding agent. 295+ audit rules across SEO, performance, security, accessibility and agent experience.
MIT19. Sept. 2026 · Metriken 2.10.0
Packagist
JayBizzle/Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
MIT27. Aug. 2026 · Metriken 2.10.0
npm
NeuraLegion/bright-cli
Command Line Interface (CLI) tool for BrightSec's solutions.
MIT11. Sept. 2026 · Metriken 2.10.0
PyPI
Liu233w/ojhunt-lite
A lightweight async Python tool for querying Online Judge (OJ) statistics across multiple platforms. Track your accepted problems (AC) and total submissions from 29+ competitive programming platforms.
BSD-2-Clause14. Sept. 2026 · Metriken 2.10.0
npm
MrAdex77/google-play-scraper
Google Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.
MIT13. Sept. 2026 · Metriken 2.10.0
npm
apmantza/pi-webaio
AiO webtools for Pi: Search, Fetch, Pull | no API keys
MIT8. Sept. 2026 · Metriken 2.10.0
PyPI
rushter/selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
MIT12. Sept. 2026 · Metriken 2.10.0
Packagist
spatie/crawler
https://spatie.be/docs/crawler
MIT12. Sept. 2026 · Metriken 2.10.0
npm
51Degrees/device-detection-node
Device detection engines for Node.js implementation of the 51Degrees Pipeline API
Eigene Lizenz9. Sept. 2026 · Metriken 2.10.0
Maven
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Apache-2.021. Okt. 2026 · Metriken 2.10.0
npm · crates.io
StopMakingThatBigFace/node-wreq
🎭 Typescript HTTP client with native TLS, HTTP2, JA3, JA4 browser impersonation backed by wreq's Rust core
MIT15. Okt. 2026 · Metriken 2.10.0
PyPI
hect0x7/JMComic-Crawler-Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
MIT15. Sept. 2026 · Metriken 2.10.0
PyPI · npm
kameleo-io/kameleo
Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.
Keine Lizenz7. Okt. 2026 · Metriken 2.10.0
NuGet
soenneker/soenneker.playwrights.crawler
A configurable Playwright crawler with rich stealth and control options.
MIT21. Okt. 2026 · Metriken 2.10.0
NuGet
JaCraig/Spidey
A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.
Apache-2.019. Okt. 2026 · Metriken 2.10.0
PyPI
PyPtt/PyPtt
The best PTT library
LGPL-3.011. Sept. 2026 · Metriken 2.10.0
PyPI
adbar/courlan
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Apache-2.012. Sept. 2026 · Metriken 2.10.0
npm
firecrawl/cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
Keine Lizenz13. Sept. 2026 · Metriken 2.10.0
PyPI
JorG18/agentcrawl
Self-hosted web scraping and Markdown extraction for AI agents
Apache-2.09. Okt. 2026 · Metriken 2.10.0
npm
cosmocoder/mcp-web-docs
Self-hosted MCP server to index and search any documentation site—public or private
MIT13. Sept. 2026 · Metriken 2.10.0
npm
iannuttall/seo
The only SEO skill your agent needs. 70+ SEO audit tools through a local CLI and MCP server, using your own crawl, Search Console, and GA4 data.
Apache-2.015. Sept. 2026 · Metriken 2.10.0
npm
just-every/mcp-read-website-fast
Quickly reads webpages and converts to markdown for fast, token efficient web scraping
MIT20. Sept. 2026 · Metriken 2.10.0
npm
qualweb/qualweb
Keine Repository-Beschreibung veröffentlicht.
ISC5. Sept. 2026 · Metriken 2.10.0
PyPI
rmax/scrapy-redis
Redis-based components for Scrapy.
MIT19. Sept. 2026 · Metriken 2.10.0
npm
rodrigogs/pornhub
pornhub API library
BSD-3-Clause18. Sept. 2026 · Metriken 2.10.0
Go
gosom/scrapemate
Golang Crawling and scraping framework
MIT15. Sept. 2026 · Metriken 2.10.0
PyPI
moskrc/crawlerdetect
🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
MIT14. Sept. 2026 · Metriken 2.10.0