Data-integratie

Event-driven, language-agnostic platform to create, schedule, and monitor workflows. In code. Coordinate data pipelines and tasks such as ETL and ELT.

A high-performance observability data pipeline.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Where data access meets operational intelligence

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Egeria core

Flink CDC is a streaming data integration tool

A curated list of awesome dbt resources

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Upserts, Deletes And Incremental Processing on Big Data.

High-performance, time-indexed object storage for robotics and industrial IoT

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Česká otevřená data

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

The Fastest Unified Streaming SQL Engine in a Single C++ Binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute real-time context from streams, logs, metrics, events, and CDC.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.

The Data Engineering Cookbook

Hop Orchestration Platform

Read and write Google Sheets as pandas DataFrames — column-matched appends, real dtypes, and layout detection for sheets that don't start at A1.

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

A Python stream processing engine modeled after Yahoo! Pipes

Clean APIs for data cleaning. Python implementation of R package Janitor

Enforce Data Contracts

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

lakeFS - Data version control for your data lake | Git for data

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

A data integration framework

Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Data Contracts engine for the modern data stack. https://www.soda.io

CLI task management & automation tool

Home of the Open Data Contract Standard (ODCS).

Insightful Tutorials and Papers about Knowledge Graphs

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Databases and Load scenarios. all defined in simple JSON or YAML — with zero coding.

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Drop-in replacement for Apache Spark UI

Fancy stream processing made operationally mundane

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Workflow Automation Engine

A curated list of awesome ETL frameworks, libraries, and software.

An Efficient RML-Compliant Engine for Knowledge Graph Construction

Pipeline open-source que baixa e processa os dados da Receita Federal para PostgreSQL

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

A curated list with resources about node-based UIs

Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨

📄 ⚙️ ETL processes for medical and scientific papers

Cloud-native, data onboarding architecture for Google Cloud Datasets

visual data prep powered by python

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Free open-source business data enrichment engine. The open-source alternative to Apollo, ZoomInfo, and Clearbit.

Making DAG construction easier

A lightweight stream processing library for Go

Logical Replication extension for PostgreSQL 17, 16, 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

Transform, query, and download geospatial data on the web.

qData is an open-source data governance and data development platform that integrates ETL, data development, metadata management, data quality, data assets, API services, and AI-powered data Q&A.

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course

📙 Awesome Data Catalogs and Observability Platforms.

Practical Data Engineering: A Hands-On Real-Estate Project Guide

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Data processing & ETL framework for Ruby

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Memphis.dev is a highly scalable and effortless data streaming platform

Example end to end data engineering project.

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Flock: A Low-Cost Streaming Query Engine on FaaS Platforms

Concurrent Python made simple

This repository is a getting started guide to Singer.

🔥 🔥 🔥Open Source & AI driven Data Onboarding Platform:Free flatfile.com alternative

Smarter data pipelines for audio.

An Awesome List of Open-Source Data Engineering Projects

Move your data with ease.

Roadmap to becoming a data engineer in 2021

Polyglot workflows without leaving the comfort of your technology stack.

**Sync 🪬 Studio**

A list of useful resources to learn Data Engineering from scratch

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Accumulated knowledge and experience in the field of Data Engineering

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.