Batch ETL using Cloud Environment which is GCP by utilizing Cloud Composer + Google Cloud Storage + Dataflow + Cloud Build

Web-scraping data engineering showcase for MMDA traffic data.

A tool for performing scheduled database backups and transferring encrypted data to secure clouds, for home labs, hobby projects, etc., in environments such as k8s, docker, vms.

Airflow powered ETL pipeline for moving Near-Earth-Object data from NASA to Google Cloud

Apache Airflow powered ETL Pipeline for moving about 133k images from Kaggle to GCS and BigQuery

Run your `pypiserver` on AppEngine with Google Cloud Storage backend ⚡

A CSV file upload and image compression system designed to handle asynchronous uploads, with status checks and webhook notifications.

Flask BoilerPlate Template for Google App Engine

This is an exercises provided by ChatGPT about sales data.

A receipt organizer (first hackathon project)

End-to-end automated modern ELT data pipeline using Python, PostgreSQL, Airflow, Kafka, GCS, BigQuery, and dbt — built to simulate a production-grade e-commerce environment. Completed as part of Purwadhika Data Engineering Bootcamp Final Project

This repo contains details about end to end implementation of the GCP GCS to BQ pipeline using CI/CD leveraging Airflow DEV and PROD Environments, Thanks

GoogleVision_OCR Project to Read out PDF

Django simple application using GAE(Google App Engine) and Cloud Storage

AWS Config Custom Resource Samples

This repository contains the files for the Discord Bot we made for the RUHacks2022 Hackathon held on May 6th, 2022

This repo contains details about how to extract API data from News API website and leveraging airflow to load the API data into GCS bucket in Parquet format and using Airflow to load the data from GCS to Snowflake Tables as needed, Thanks

Implementation of a standard file system with POSIX interface that uses Google Cloud storage object