Data-integratie

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

Event-driven, language-agnostic platform to create, schedule, and monitor workflows. In code. Coordinate data pipelines and tasks such as ETL and ELT.

Platform to programmatically author, schedule, and monitor workflows.

A high-performance observability data pipeline.

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Where data access meets operational intelligence

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Collect, unify, transform, and store your customer data, and route it to a wide range of common, popular marketing, sales, and product tools (alternative to Segment).

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Scalable and efficient data transformation framework - backwards compatible with dbt.

Turns Data and AI algorithms into production-ready web applications in no time.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Flink CDC is a streaming data integration tool

Unified MySQL, Postgres & FlightSQL Server, Powered by DuckDB.

High-performance, time-indexed object storage for robotics and industrial IoT

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Incremental engine for long horizon agents 🌟 Star if you like it!

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Upserts, Deletes And Incremental Processing on Big Data.

The best place to learn data engineering. Built and maintained by the data engineering community.

TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

A comprehensive Python package template to kickstart and standardize your MLOps initiatives and data pipelines.

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.

Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync

Egeria core

数据流引擎是一款面向数据集成、数据同步、数据交换、数据共享、任务配置、任务调度的底层数据驱动引擎。数据流引擎采用管执分离、多流层、插件库等体系应对大规模数据任务、数据高频上报、数据高频采集、异构数据兼容的实际数据问题。

Maestro: Netflix’s Workflow Orchestrator

ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Hop Orchestration Platform

ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases

Česká otevřená data

MacOS app written in Swift that bulk exports Apple Notes (including iCloud Notes) to a multitude of formats preserving note folder structure.

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

An orchestration platform for the development, production, and observation of data assets.

Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 20+ RDBMS and NoSQL data sources.

Enforce Data Contracts

Read and write Google Sheets as pandas DataFrames — column-matched appends, real dtypes, and layout detection for sheets that don't start at A1.

A Spark plugin for reading and writing Excel files

Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI

Home of the Open Data Contract Standard (ODCS).

A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.

A Python stream processing engine modeled after Yahoo! Pipes

Monopoly is a Python library & CLI that converts bank statement PDFs to CSV.

Clean APIs for data cleaning. Python implementation of R package Janitor

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

loads CSV, TSV, LTSV, JSON, JSONL, Parquet, XLSX, ACH, and Fedwire files into SQLite; includes prep and frame for cleanup and in-memory transforms

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

A dbt adapter for Databricks.

Drop-in replacement for Apache Spark UI

A data integration framework

Data Contracts engine for the modern data stack. https://www.soda.io

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Databases and Load scenarios. all defined in simple JSON or YAML — with zero coding.

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Use LLMs to robustly extract web data

The Common Core Ontology Repository holds the current released version of the Common Core Ontology suite.

Snowflake Snowpark Python API

The Trino (https://trino.io/) adapter plugin for dbt (https://getdbt.com)

dbt adapter for SQL Server and Azure SQL

lakeFS - Data version control for your data lake | Git for data

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

The data-validation toolkit for enhanced dbt (data build tool) PR review

DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.

Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.

In-memory Java DataFrame library

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

CLI tool for dbt users to simplify creation of staging models (yml and sql) files

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

TAK Compatible, browser based Common Operation Picture & Situational Awareness tool

Blazing-fast Data-Wrangling toolkit

大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批流,私域营销等模块

Open Source Feature Flags, Experimentation, and Product Analytics

Workflow Automation Engine

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.

Powerful RDF knowledge graph generation with RML mappings

SDK for robotics teams to verify the quality of their data used for AI model training.

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.

A collection of AI skills for working with Dagster

🎼 Integrate multiple high-dimensional datasets with fuzzy k-means and locally linear adjustments.

mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.

sync/async iterable streams for Python

Fancy stream processing made operationally mundane

A curated collection of 300+ engineering blog articles from top tech companies. Learn how the best engineering teams solve real-world problems at scale.

Agentic Data Engineering Harness for building data pipelines, data products, data APIs, and data lakes autonomously

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

superglue (YC W25) builds integrations and tools from natural language. Get production-grade tools for long tail and enterprise systems.

A 股、期货、期权数据本地化管道：Tushare Pro 拉取 → Parquet 分区存储 → DuckDB 本地查询，支持增量同步与定时调度

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Server application to serve U.S. federal spending data via a RESTful API

中国电商 MCP 连接器：8 个平台、155 个已注册工具。开源标准版，含能力矩阵与实际工程验收记录；真实商家联调状态逐项公开。Chinese commerce MCP connectors with documented capabilities and validation status.

Relational Workflows: where database schemas define executable data pipelines.

The leader in Customer Data Infrastructure

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

Beginner data engineering project - batch edition

End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.

ultra low latency graph based stream processing framework

📄 ⚙️ ETL processes for medical and scientific papers

Pipeline open-source que baixa e processa os dados da Receita Federal para PostgreSQL

Pentaho Data Integration ( ETL ) a.k.a Kettle

SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.

visual data prep powered by python

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

A cross-platform command line tool for parallelised content extraction and analysis.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

Work with your web service, database, and streaming schemas in a single format.

Cloud-native, data onboarding architecture for Google Cloud Datasets

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Projects done in the Data Engineering Nanodegree by Udacity.com

go-etl is a toolset for data extraction, transformation and loading.

An open source, standard data file format for graph data storage and retrieval.

Make dbt great again! Extend dbt with plugins, local docs and custom adapters — fast, safe, and developer-friendly

A lightweight workflow engine

Production-grade PostgreSQL extension to execute arbitrary SQL in background worker processes — with async execution, autonomous transactions, cookie-protected handles, cancellation, progress reporting, and observability.

Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources

Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.

end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck

The premier open source Data Quality solution

Yet another redundant workflow engine

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Logical Replication extension for PostgreSQL 17, 16, 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

qData is an open-source data governance and data development platform that integrates ETL, data development, metadata management, data quality, data assets, API services, and AI-powered data Q&A.

Making DAG construction easier

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course

Streaming reactive and dataflow graphs in Python

Radient turns many data types (not just text) into vectors for similarity search, RAG, regression analysis, and more.

A lightweight stream processing library for Go

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

A simplified, lightweight ETL Framework based on Apache Spark

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch

NicheNet: predict active ligand-target links between interacting cells

Practical Data Engineering: A Hands-On Real-Estate Project Guide

Dagster Labs' open-source data platform, built with Dagster.

Code and data for the Modern Polars book

A collection of tools for extracting FHIR resources and analytics services on top of that data.

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

🥪🦘 An open source sandbox project exploring dbt workflows via a fictional sandwich shop's data.

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Open Source Data Quality Monitoring.

The most comprehensive SQL guide from a real-world expert! Learn everything from basics to advanced queries, optimizations, and real-world SQL

PyAirbyte brings the power of Airbyte to every Python developer. Powers the Airbyte Cloud Replication MCP.

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

Categorical Query Language IDE

Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.

OHLC,news,economic event restfull api and WebSocket API for specifically designed for AI quantitative trading/training.Multiple Timeframes,Multiple-Symbols-Multiple-Timeframes

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Code for "Efficient Data Processing in Spark" Course

The Open-Source Enterprise Data Platform in a single Portal

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee

Data processing & ETL framework for Ruby

A utility library for comparing and synchronizing different datasets.

A list about Apache Kafka

汇总Apache Hudi相关资料

A tool for building feature stores.

Power of appbase.io via CLI, with nifty imports from your favorite data sources

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Memphis.dev is a highly scalable and effortless data streaming platform

Example end to end data engineering project.

Fluent data pipelines for python and your shell

Airbyte made simple (no UI, no database, no cluster)

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Concurrent Python made simple

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.

🔥 🔥 🔥Open Source & AI driven Data Onboarding Platform:Free flatfile.com alternative

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Fast, sensitive and accurate integration of single-cell data with Harmony

A simple Spark-powered ETL framework that just works 🍺

This repository is a getting started guide to Singer.

Smarter data pipelines for audio.

Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Flock: A Low-Cost Streaming Query Engine on FaaS Platforms

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

A lightweight CLI tool for versioning data alongside source code and building data pipelines.

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

A visual process builder

Example repository showing how to build a data platform with Prefect, dbt and Snowflake

Polyglot workflows without leaving the comfort of your technology stack.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!