#

etl

Here are 2,932 public repositories matching this topic...

pathway

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

Updated Apr 4, 2026
Python

airflow

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Updated Apr 4, 2026
Python

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Updated Apr 4, 2026
Python

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

Updated Apr 3, 2026
Python

mage-ai / mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

Updated Apr 2, 2026
Python

aws-sdk-pandas

aws / aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Updated Apr 4, 2026
Python

ucbepic / docetl

A system for agentic LLM-powered data processing and ETL

python workflow data etl semantic-data elt data-pipelines agents document-analysis document-processing unstructured-data unstructured-data-analysis llm

Updated Mar 27, 2026
Python

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

export bigquery aws csv sql etl ethereum transaction gcp google-cloud erc20 erc20-tokens blockchain-analytics erc721

Updated Jan 25, 2026
Python

SQLMesh / sqlmesh

Scalable and efficient data transformation framework - backwards compatible with dbt.

python sql etl dataops dbt elt transformation dataengineering

Updated Apr 3, 2026
Python

instill-ai / instill-core

🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications

python api cli golang open-source typescript ai pipeline etl developer-tools gpt hacktoberfest low-code no-code unstructured-data llm stable-diffusion generative-ai

Updated Apr 4, 2026
Python

AlexIoannides / pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

python data-science spark etl pyspark data-engineering etl-pipeline etl-job

Updated Jan 1, 2023
Python

mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

python data pipeline etl postgresql data-integration

Updated Dec 15, 2023
Python

nerevu / riko

A Python stream processing engine modeled after Yahoo! Pipes

cli rss data library etl functional-programming asynchronous parallelism stream-processing featured

Updated Dec 28, 2021
Python

superlinked / sie

Superlinked Inference Engine is an Open-source inference server and production cluster for embeddings, reranking, and extraction.

python nlp natural-language-processing information-retrieval deep-learning etl retrieval ml embeddings vectorization semantic-search data-pipeline mlops vector-search vector-database llm retrieval-augmented-generation

Updated Apr 2, 2026
Python

toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync

mysql python elasticsearch kibana sql etl postgresql mariadb mysql-database psycopg2 pymysql change-data-capture opensearch elasticsearch-sync

Updated Mar 10, 2026
Python

Open-Source-Legal / OpenContracts

Humans and AI agents, building knowledge bases together. Self-hosted document annotation, version control, semantic search, and MCP.

agent etl unstructured-data etl-pipeline vector-database llm prompt-engineering agentic-ai

Updated Apr 4, 2026
Python

NeumAI

NeumTry / NeumAI

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

python data database ai ops pipeline etl retrieval embeddings data-engineering vectors rag mlops vector-database llm chatgpt llmops

Updated Jan 15, 2024
Python

stitchfix / hamilton

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

python data-science machine-learning etl numpy pandas data-engineering data-platform software-engineering feature-engineering dataframe dag hamiltonian etl-framework hamilton featurization etl-pipeline stitch-fix

Updated Jul 3, 2023
Python

eland

elastic / eland

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

python elasticsearch machine-learning big-data etl scikit-learn pandas lightgbm data-analysis dataframe dataframes time-series-forecasting eland

Updated Apr 1, 2026
Python

paperetl

neuml / paperetl

📄 ⚙️ ETL processes for medical and scientific papers

python parse etl medical scientific-papers

Updated Dec 7, 2025
Python

Improve this page

Add a description, image, and links to the etl topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the etl topic, visit your repo's landing page and select "manage topics."