web-crawler — Gesundheit von Open-Source-Software

npm

99AußergewöhnlichGesundheitsindex

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

TypeScript · MDX★ 26.2K↓ 6.8M/Monat19. Okt. 2026

Apache-2.019. Okt. 2026 · Metriken 2.10.0

PyPI · npm

99AußergewöhnlichGesundheitsindex

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Python · MDX★ 9.535↓ 975.8K/Monat17. Sept. 2026

Apache-2.017. Sept. 2026 · Metriken 2.10.0

PyPI

94AußergewöhnlichGesundheitsindex

D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ

Python★ 81.4K16. Sept. 2026

BSD-3-Clause16. Sept. 2026 · Metriken 2.10.0

PyPI

94AußergewöhnlichGesundheitsindex

ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

Python★ 31K16. Sept. 2026

MIT16. Sept. 2026 · Metriken 2.10.0

Maven

94AußergewöhnlichGesundheitsindex

apache/stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

Java★ 1.00819. Okt. 2026

Apache-2.019. Okt. 2026 · Metriken 2.10.0

PyPI · Packagist · crates.io +2

94AußergewöhnlichGesundheitsindex

firecrawl/firecrawl

The context API to search, scrape, and interact with the web at scale. 🔥

TypeScript · Python★ 181.1K↓ 5.8M/Monat16. Sept. 2026

AGPL-3.016. Sept. 2026 · Metriken 2.10.0

crates.io · npm · PyPI

91ExzellentGesundheitsindex

fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Rust★ 1.095↓ 10.6K/Monat2. Okt. 2026

AGPL-3.02. Okt. 2026 · Metriken 2.10.0

crates.io · npm · PyPI

91ExzellentGesundheitsindex

us/crw

Rust★ 1.032↓ 10.7K/Monat16. Sept. 2026

AGPL-3.016. Sept. 2026 · Metriken 2.10.0

npm

89ExzellentGesundheitsindex

JustinBeckwith/linkinator

Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.

TypeScript★ 1.269↓ 691.2K/Monat3. Okt. 2026

MIT3. Okt. 2026 · Metriken 2.10.0

npm

89ExzellentGesundheitsindex

firecrawl/firecrawl-mcp-server

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

TypeScript · JavaScript★ 7.332↓ 492.7K/Monat28. Aug. 2026

MIT28. Aug. 2026 · Metriken 2.10.0

npm · PyPI

89ExzellentGesundheitsindex

webrecorder/browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

TypeScript★ 1.11724. Aug. 2026

AGPL-3.024. Aug. 2026 · Metriken 2.10.0

npm · crates.io

87ExzellentGesundheitsindex

0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Rust★ 2.343↓ 1.301/Monat13. Sept. 2026

AGPL-3.013. Sept. 2026 · Metriken 2.10.0

npm

84ExzellentGesundheitsindex

KnockOutEZ/wigolo

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

TypeScript★ 5.237↓ 3.896/Monat12. Sept. 2026

Eigene Lizenz12. Sept. 2026 · Metriken 2.10.0

crates.io

84ExzellentGesundheitsindex

spider-rs/spider

Foundational low latency web data collecting in Rust

Rust★ 2.790↓ 24.2K/Monat20. Okt. 2026

MIT20. Okt. 2026 · Metriken 2.10.0

PyPI

84ExzellentGesundheitsindex

tobocop2/lilbee

The whole local AI stack in one executable: it runs and manages local AI models across every GPU, and it's a search engine you can talk to, with cited answers from your files, code, and the web. MCP server for coding agents, web crawler, TUI, CLI, REST API, Python library. No Ollama or LM Studio needed, works with both.

Python★ 54↓ 3.110/Monat18. Sept. 2026

MIT18. Sept. 2026 · Metriken 2.10.0

PyPI · npm

83ExzellentGesundheitsindex

ScrapeGraphAI/scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

Jupyter Notebook★ 86↓ 212.9K/Monat7. Sept. 2026

MIT7. Sept. 2026 · Metriken 2.10.0

80ExzellentGesundheitsindex

kinorai/omnifeed

LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible

Go · HTML★ 1021. Okt. 2026

MIT21. Okt. 2026 · Metriken 2.10.0

npm · PyPI · crates.io

78GutGesundheitsindex

Goldziher/basemind

Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

Rust★ 99↓ 2.971/Monat14. Sept. 2026

MIT14. Sept. 2026 · Metriken 2.10.0

npm

78GutGesundheitsindex

scrapfly/typescript-scrapfly

Official TypeScript/JavaScript SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Ships to npm, JSR, and Deno.

TypeScript★ 2317. Okt. 2026

Eigene Lizenz17. Okt. 2026 · Metriken 2.10.0

npm · Packagist · crates.io +1

78GutGesundheitsindex

xberg-io/crawlberg

High-performance web crawling engine with bindings for 11 languages

Rust★ 173↓ 1.094/Monat15. Sept. 2026

MIT15. Sept. 2026 · Metriken 2.10.0

Maven

77GutGesundheitsindex

Norconex/crawler

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

Java★ 20521. Okt. 2026

Apache-2.021. Okt. 2026 · Metriken 2.10.0

PyPI

73GutGesundheitsindex

JorG18/agentcrawl

Self-hosted web scraping and Markdown extraction for AI agents

Python★ 10↓ 2.758/Monat9. Okt. 2026

Apache-2.09. Okt. 2026 · Metriken 2.10.0

73GutGesundheitsindex

jonhadfield/ip-fetcher

Go library and CLI to fetch public IP ranges from cloud platforms, CDNs, crawler bots, monitoring probes and threat intelligence feeds

Go★ 412. Sept. 2026

MIT12. Sept. 2026 · Metriken 2.10.0

69GutGesundheitsindex

gosom/scrapemate

Golang Crawling and scraping framework

Go★ 21015. Sept. 2026

MIT15. Sept. 2026 · Metriken 2.10.0

npm

69GutGesundheitsindex

mysleekdesigns/crawlforge-mcp

31 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.

JavaScript★ 2↓ 5.464/Monat21. Okt. 2026

MIT21. Okt. 2026 · Metriken 2.10.0

PyPI · npm · crates.io +1

62MittelGesundheitsindex

spider-rs/spider-clients

Python, Javascript, and Rust libraries for the Spider Cloud API.

Rust · Python · Go★ 26↓ 365K/Monat9. Sept. 2026

MIT9. Sept. 2026 · Metriken 2.10.0

Go · RubyGems

60MittelGesundheitsindex

1mb-dev/lobster

Intelligent web stress testing with auto URL discovery. Crawls your app, validates performance under load, generates actionable reports. Zero config required.

Go★ 219. Sept. 2026

MIT19. Sept. 2026 · Metriken 2.10.0

npm

38SchwachGesundheitsindex

Aegis-Runner/AegisRunner

AegisRunner crawls any website from a single URL, discovers every page, form, and interaction, then uses AI to generate a complete Playwright test suite. Also runs accessibility, SEO, security, and performance audits on every page — no recording, no scripting, no setup.

JavaScript★ 1↓ 2.534/Monat13. Sept. 2026

MIT13. Sept. 2026 · Metriken 2.10.0

Maven

23GefährdetGesundheitsindex

ssssssss-team/spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Java★ 11.4K27. Aug. 2026

MIT27. Aug. 2026 · Metriken 2.10.0

#web-crawler