#crawler

A lightweight async Python tool for querying Online Judge (OJ) statistics across multiple platforms. Track your accepted problems (AC) and total submissions from 29+ competitive programming platforms.

Google Play scraper for Node.js with a fully typed TypeScript API. App details, search, top charts, reviews, permissions and data safety. A modern replacement for the unmaintained google-play-scraper.

AiO webtools for Pi: Search, Fetch, Pull | no API keys

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

https://spatie.be/docs/crawler

Device detection engines for Node.js implementation of the 51Degrees Pipeline API

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

🎭 Typescript HTTP client with native TLS, HTTP2, JA3, JA4 browser impersonation backed by wreq's Rust core

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.

A configurable Playwright crawler with rich stealth and control options.

A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.

The best PTT library

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents

Self-hosted web scraping and Markdown extraction for AI agents

Self-hosted MCP server to index and search any documentation site—public or private

The only SEO skill your agent needs. 70+ SEO audit tools through a local CLI and MCP server, using your own crawl, Search Console, and GA4 data.

Quickly reads webpages and converts to markdown for fast, token efficient web scraping

Опис репозиторію не опубліковано.

Redis-based components for Scrapy.

pornhub API library

Golang Crawling and scraping framework

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.