A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.

An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.

End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)

A python application that leverages Microsoft Azure to build a fully functional trading system.

Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.

Formula 1 race data engineering project which utilises azure services and databricks to ingest and analyse the data.

Data engineering project using Azure Data Factory and Apache Airflow

A demand forecasting pipeline deployed on Azure and AWS

A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.

Python script that cleans the JSON document used by an Azure Data Factory or Azure Synapse Copy Activity to specify the source-sink mapping.

This project demonstrates the development of a complete end-to-end data pipeline in the cloud using the Microsoft Azure ecosystem to ingest, transform, store, and visualize COVID-19 related data.

In this project, I've created an end-to-end ETL pipeline and subsequently developed a machine learning model to predict the price of Amazon products based on several product-related features.

This project uses Databricks and Microsoft Azure (mostly) to deploy a complete data processing workflow, covering key concepts such as ETLs, data pipelines, machine learning, and data visualization.

Projeto utilizando a base de CNPJ da Receita Federal

This project demonstrates a comprehensive data engineering pipeline capable of transforming vast amounts of historical F1 data into valuable insights. By leveraging modern cloud-based technologies and automating key processes, the project is designed to be both scalable and adaptable to future requirements.

Parse Azure Data Factoty ETL pipeline to json

Production-grade customer segmentation pipeline built on Azure (Blob Storage, Data Factory, Azure ML, Batch Endpoint). Includes end-to-end data engineering, feature engineering, K-Means model training, and scalable batch inference.

ETL pipeline tailored for Olympics data

Treinamento Azure Data Factory + Azure Databricks + Azure Analysis Services + Power BI

Repository for the Visual TOM to Azure Data Factory connector