Packagist
mvdbos/php-spider
A configurable and extensible PHP web spider
MIT19 жовт. 2026 р. · метрики 2.10.0
Усі теги
Тег каталогу
#crawler
Усі репозиторії публічного реєстру з цим тегом — із тем GitHub або ключових слів, які публікують їхні реєстри пакетів. Здоров'я вимірюється за тією ж версіонованою методологією, що й решта реєстру.
114 записів
Packagist
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
MIT16 вер. 2026 р. · метрики 2.10.0
PyPI
scrapy-plugins/scrapy-zyte-smartproxy
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
BSD-3-Clause20 вер. 2026 р. · метрики 2.10.0
Go
x-way/crawlerdetect
Golang module to detect bots and crawlers via the user agent
MIT9 вер. 2026 р. · метрики 2.10.0
Go
krau/ManyACG
Collect, Download, Organize and Share your Favorite Anime Artworks.
AGPL-3.020 жовт. 2026 р. · метрики 2.10.0
NuGet
zzzprojects/html-agility-pack
Html Agility Pack (HAP) is a free and open-source HTML parser written in C# to read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files.
MIT20 жовт. 2026 р. · метрики 2.10.0
Go · npm
Autumn-27/ScopeSentry
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
Без ліцензії18 вер. 2026 р. · метрики 2.10.0
npm
ODATANO/NIGHTGATE
A CAP-based OData V4 gateway to the Midnight privacy chain, providing confidential on-chain data access and privacy-preserving transaction execution to any enterprise system
Apache-2.010 вер. 2026 р. · метрики 2.10.0
RubyGems
loadkpi/crawler_detect
Ruby gem to detect bots and crawlers via the user agent
MIT6 вер. 2026 р. · метрики 2.10.0
npm
mysleekdesigns/crawlforge-mcp
28 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
MIT5 вер. 2026 р. · метрики 2.10.0
Go
uinaf/lincrawl
Local-first Linear work-graph archive CLI
MIT21 лип. 2026 р. · метрики 2.10.0
PyPI · npm · crates.io +1
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
MIT9 вер. 2026 р. · метрики 2.10.0
npm
codepurse/SEOCORE
Enterprise-grade, multi-threaded SEO Crawler, Rule Engine, and Link Graph Analyzer. Built in TypeScript for speed, compliance, and deep site health audits.
MIT9 вер. 2026 р. · метрики 2.10.0
PyPI
gabrielguarisa/brdata
Brazilian financial market data sources
Без ліцензії19 вер. 2026 р. · метрики 2.10.0
Maven
fanyong920/jvppeteer
Java API For Chrome and Firefox
Apache-2.010 вер. 2026 р. · метрики 2.10.0
npm
olostep-api/olostep-js
nodejs SDK of the Olostep API
Без ліцензії15 вер. 2026 р. · метрики 2.10.0
npm
brandonkramer/pi-scraper
Pi extension for fast page scraping, recursive crawling, URL/site mapping, brand extraction, content diffing, PDF text extraction, and deterministic vertical extraction.
MIT20 жовт. 2026 р. · метрики 2.10.0
PyPI
codelucas/newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
MIT17 вер. 2026 р. · метрики 2.10.0
Packagist
JayBizzle/Laravel-Crawler-Detect
A Laravel wrapper for CrawlerDetect - the web crawler detection library
MIT14 вер. 2026 р. · метрики 2.10.0
Go
naiba/proxy-in-a-box
Automatic proxy pool for web scraping - crawls, validates, and rotates proxies with rate limiting and MITM support
MIT19 жовт. 2026 р. · метрики 2.10.0
npm
Kikobeats/top-crawler-agents
Curated list of common web crawler and bot User-Agent strings used across the internet.
MIT19 вер. 2026 р. · метрики 2.10.0
Maven · npm
seart-group/ghs
GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them
MIT19 вер. 2026 р. · метрики 2.10.0
Go
neumachen/gojira
Jira issue graph CLI and Go toolkit — fetch, crawl, mirror, and serve Jira issues with their links, PRs, and relationships
MIT19 вер. 2026 р. · метрики 2.10.0
PyPI
tobybgy-lsd/web-agent-runtime-bench
Local-first failure diagnosis, auto collection, repair planning, AI handoff, and verification for Playwright, crawler, RPA, and agent workflows.
Власна ліцензія13 вер. 2026 р. · метрики 2.10.0
NuGet
Laraue/Laraue.Crawling
The set of tools for fast writing crawlers on the .NET.
MIT4 жовт. 2026 р. · метрики 2.10.0
Packagist
jae-jae/QueryList
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Без ліцензії15 вер. 2026 р. · метрики 2.10.0
Go
mjc-gh/virgo
Convert a webpage into plaintext or markdown using the Chrome DevTool Protocol
BSD-3-Clause10 вер. 2026 р. · метрики 2.10.0
Go
quantmind-br/repodocs
Go CLI for extracting websites, repositories, sitemaps, and package docs into structured Markdown
MIT25 серп. 2026 р. · метрики 2.10.0
Packagist · npm
duzun/hQuery.php
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
MIT14 вер. 2026 р. · метрики 2.10.0
npm
webscraping-ai/webscraping-ai-mcp-server
A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.
Без ліцензії12 вер. 2026 р. · метрики 2.10.0