#data-engineering

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Apache Superset is a Data Visualization and Data Exploration Platform

Workflow Engine for Kubernetes

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

The data-validation toolkit for enhanced dbt (data build tool) PR review

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

An orchestration platform for the development, production, and observation of data assets.

The Open Source Feature Store for AI/ML

Always know what to expect from your data.

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Privacy and Security focused Segment-alternative, in Golang and React

Enforce Data Contracts

Open Source Feature Flags, Experimentation, and Product Analytics

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

Python Streaming DataFrames for Kafka

Snowflake Snowpark Python API

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

A MCP (Model Context Protocol) server for interacting with dbt.

Visualise your Kedro data and machine-learning pipelines and track your experiments.

Fancy stream processing made operationally mundane

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Clean APIs for data cleaning. Python implementation of R package Janitor

lakeFS - Data version control for your data lake | Git for data

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.