#data-engineering

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Market data acquisition, storage, and update workflows for machine learning for trading.

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

Relational Workflows: where database schemas define executable data pipelines.

Workbench: An easy to use Python API for creating and deploying AWS SageMaker Models

PyAirbyte brings the power of Airbyte to every Python developer. Powers the Airbyte Cloud Replication MCP.

Brain of Bolster, Be Ware

Apache Airflow-compatible workflow orchestrator, re-engineered in Go — the drop-in Airflow UI + REST API, but on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered.

Config-driven data ingestion and SCD2 historization framework built on dlt

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

Datos públicos de Chile curados, normalizados y validados: geografía, demografía, economía, salud, educación y distritos electorales. Listos para consumir en una línea de código con Polars, DuckDB, SQLite y Excel.

Inspection of tabular (csv, xls-like, parquet) files to guess the columns' content

Dex is the agent-native analytics engineering toolkit. Point it at your warehouse and your dbt project. It learns the landscape, authors your transformations, and tells you exactly what to fix when the schema drifts. Built for analytics engineers and data engineers who want more out of their coding agent.

Datagrunt is a Python library designed to simplify the way you work with CSV, Excel, Parquet, and PDF files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.

Synthetic data that hits the numbers you declare, exactly. Multi-table with verified foreign-key integrity, deterministic, no model in the data path. Python + MCP server. In simple terms, a powerful demo data generator for sales/demos/seed data.

Frontend & BFF (Backend for frontend) for Olake. This includes the UI code and backend code for storing the configuration of sync and orchestrating it.

Data Engineering Toolbelt - DB-to-Glass

Control Plane For Data Workflow With Built-In Agentic Remediation

The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.