A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.

An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.

End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)

A demand forecasting pipeline deployed on Azure and AWS

A python application that leverages Microsoft Azure to build a fully functional trading system.

Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.

Azure End To End Data Engineering Project | Azure Data Factory | Azure Databricks | Azure SQL DB | PySpark | Big Data. It is a in depth Data Engineering project using powerful tools like Azure Data Factory, Azure SQL DB, Azure Databricks, Unity Catalog, Delta Live Tables, Spark Streaming, PySpark, Databricks Asset Bundles, GitHub, and more.

A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.

Using LLMS to extract medical Data for Doctors and Hospitals

Treinamento Azure Data Factory + Azure Databricks + Azure Analysis Services + Power BI

Azure Data Factory v2 datapipeline sample code snippets...

Parse Azure Data Factoty ETL pipeline to json

Production-grade customer segmentation pipeline built on Azure (Blob Storage, Data Factory, Azure ML, Batch Endpoint). Includes end-to-end data engineering, feature engineering, K-Means model training, and scalable batch inference.

This branch focuses on building Data Engineering Interview Question and Answer

RideStream is a scalable Azure-based lakehouse project designed for ride-hailing analytics. It combines batch ingestion from HTTP/internal sources with streaming booking events from Event Hubs, processes them in Databricks, and delivers a clean analytics model through a silver-layer OBT and a gold-layer star schema.

End-to-end Azure Data Engineering project using ADF for incremental ingestion, Databricks (DLT) for Medallion Architecture, and Delta Lake for CDC (SCD Type 1). Managed via Databricks Asset Bundles (DABs) for professional CI/CD. Focuses on real-time streaming, scalability, and Star Schema modeling.

This project demonstrates a comprehensive data engineering pipeline capable of transforming vast amounts of historical F1 data into valuable insights. By leveraging modern cloud-based technologies and automating key processes, the project is designed to be both scalable and adaptable to future requirements.

ETL pipeline tailored for Olympics data

Data pipeline that processes Formula1 data with Azure Databricks, DeltaLake, and Azure Data Factory