Beginner data engineering project - batch edition

📘 Making Data, the DataMade Way

Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.

Nextract is a Extract Transform Load (ETL) platform build on top of Node.js streams

Ethereum Analytical Database - Ethereum data access solution that can be used for analytics and application development. The solution works on a fast DB - Clickhouse.

dbt tutorial using a local PostgreSQL database

Wurzel is an open-source Python framework for advanced ETL pipelines in Retrieval-Augmented Generation (RAG) systems.

Integrating Apache Airflow, dbt, Great Expectations and Apache Superset to develop a modern open source data stack.

Socialytics is a social network data acquisition (ETL), analytics and intelligence tool.

Formação de Data Analytics - Turmas Alex Souza

In this project, dbt, Great Expectations, Python and Pandas were used to transform and validate the "Inside Airbnb" dataset. The tools ensure quality data, ready for analysis.

Solo-built a senior-level analytics platform in 5 weeks: 471 messy government files → 320+ tested dbt models → 18 marts with 100+ CFO & equity KPIs → public BigQuery + 3 boardroom-ready Power BI dashboards. From zero knowledge to production-grade warehouse in 4 months — core build done in 5 weeks

Tutorial using Great Expectations library, validating and profiling data on a local PostgreSQL database.

Node-RED ndoes for streaming data - utilising NodeJS Stream APIs

An open-source archive of campaign finance and lobbying disclosure data from the California Secretary of State’s CAL-ACCESS database

NiFi, Data Engineering, Data Ingest, REST, ETL, Mapping, ELT, SQL, Spark, Kafka for Good

ETL oriented Celery MongoDB skeleton framework

In this project, we built a database that demonstrates the changes in American top fastest-growing private companies through time. The database is built on by ingesting, combining, and restructuring data from three main data sources into a conformed one Postgresql database, and deploy into the Flask app.