data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
-
Updated
Sep 21, 2026 - Python
#
data-lake
Here are 624 public repositories matching this topic...
lakeFS - Data version control for your data lake | Git for data
go golang apache-spark aws-s3 google-cloud-storage data-engineering data-lake azure-storage data-version-control object-storage datalake hadoop-filesystem data-quality data-versioning azure-blob-storage apache-sparksql git-for-data lakefs datalakes
-
Updated
Sep 17, 2026 - Go
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
-
Updated
Sep 20, 2026 - Scala
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
infrastructure aws postgres data airflow cloudformation cassandra cluster aws-s3 aws-sdk data-warehouse data-engineering data-lake aws-ec2 postgresql-database data-modeling cassandra-database etl-pipeline data-engineering-pipeline airflow-operators
-
Updated
Aug 26, 2022 - Python
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
real-time big-data high-performance data-lake data-integration flink data-synchronization data-pipeline
-
Updated
Jan 1, 2024 - Java
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
Apache Iceberg REST Catalog in Rust — access control, credential vending and audit for every engine and AI agent. Apache 2.0.
rust catalog data-lake rbac access-control iceberg data-governance apache-iceberg lakehouse openfga open-lakehouse lakehouse-governance iceberg-rest-catalog
-
Updated
Sep 21, 2026 - Rust
Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.
big-data spark data-lake flink management-system iceberg trino hudi paimon lake-house self-optimizing
-
Updated
Sep 20, 2026 - Java
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
-
Updated
Jan 12, 2023 - Java
Personal Data Engineering Projects
postgres airflow spark cassandra mongodb data-warehouse data-engineering data-lake scrapy data-modeling aws-redshift star-schema ingest-data data-engineering-nanodegree
-
Updated
Feb 8, 2023 - Jupyter Notebook
An efficient storage and compute engine for both on-prem and cloud-native data analytics.
-
Updated
Sep 18, 2026 - Java
Data API Framework for AI Agents and Data Apps
bigquery typescript sql database ai analytics clickhouse reporting postgresql spreadsheet snowflake data-warehouse data-lake restful-api api-builder ksqldb duckdb ai-agent vulcan-sql vulcansql
-
Updated
Jul 1, 2024 - TypeScript
This repository has been merged into Canner/WrenAI under the core/ directory
agent semantic data sql ai mcp data-warehouse data-lake business-intelligence data-analytics data-analysis hacktoberfest semantic-layer llm agentic-ai mcp-server
-
Updated
May 6, 2026 - Java
Enterprise-grade, production-hardened, serverless data lake on AWS
-
Updated
Sep 4, 2026 - Python
Generic Data Ingestion & Dispersal Library for Hadoop
-
Updated
Mar 19, 2023 - Java
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
python java kubernetes mqtt cloud kafka mongodb tensorflow terraform gcp grpc data-lake confluent hivemq kafka-connect kafka-streams ksql ksqldb tiered-storage tensorflow-io
-
Updated
Nov 5, 2020 - Jupyter Notebook
GigAPI is a Timeseries lakehouse for real-time data and sub-second queries, powered by DuckDB OLAP + Parquet Query Engine, Compactor w/ Cloud-Native Storage. Drop-in FDAP alternative ⭐
api golang sql database rest-api s3 data-lake olap parquet query-engine datalake clickhouse-server lakehouse duckdb qryn gigapipe duckdb-api fdap duckdb-server ducklake
-
Updated
Oct 20, 2025 - Go
Use SQL to build ELT pipelines on a data lakehouse.
sql apache-spark etl pipelines data-engineering data-lake data-transfer delta data-integration upsert elt data-pipeline datalake data-ingestion spark-sql zeppelin-notebook apache-iceberg lakehouse incremental-updates
-
Updated
May 25, 2022 - JavaScript
BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)
-
Updated
Apr 7, 2025 - C++
中国 A 股数据基础设施。42 个日更数据集：行情、基本面、资金面、公告事件、指数行业、宏观与风险。行级溯源、PIT 语义、复权与历史成分内置，MCP 原生。自托管，零注册、零 API Token
python mcp fintech data-lake stock-market market-data parquet quantitative-finance financial-data tushare backtesting point-in-time a-shares akshare duckdb ai-agent polars china-stock-market llm-tools china-equity
-
Updated
Sep 19, 2026 - Python
Add this topic to your repo
To associate your repository with the data-lake topic, visit your repo's landing page and select "manage topics."