-
Updated
Oct 15, 2020 - Java
#
gcp-dataflow
Here are 27 public repositories matching this topic...
GCP Space Shepherd - service for monitoring Google DataFlow executions
-
Updated
Oct 17, 2025 - Java
This repo is dedicated for GCP data engineering concepts: BigTable, BigQuery, DataFlow, PubSub, DataProc Spark on GCP. Apache Beam, Apache AirFlow
-
Updated
Oct 13, 2020 - Java
This project focuses on scalable data processing and query performance optimisation. It uses Snowflake for data warehousing, GCP Cloud Functions for serverless compute, and Apache Kafka for real-time data streaming. It leverages the serverless capabilities of the systems for scalability and performance.
performance scalability velocity snowflake volume apache-kafka auto-scaling query-optimization apache-airflow gcp-dataflow
-
Updated
Sep 10, 2024 - HCL
GCP Streaming Data Pipeline for Building Energy Consumption
-
Updated
Feb 18, 2020 - Python
-
Updated
Apr 28, 2021 - Go
Black Friday, the biggest shopping day of the year, presents a unique opportunity for retailers like Walmart to boost sales, attract new customers, and clear inventory. Managing the surge in transaction volumes, understanding customer preferences, and optimizing inventory in real time are critical challenges that require sophisticated data solution
-
Updated
Feb 4, 2024 - Python
Github action to create dataflow templates
-
Updated
Aug 22, 2024 - Shell
GCP Dataflow pipeline with BigQuery as source and side input
-
Updated
Aug 9, 2018 - Python
Sample projects to explore various Google Cloud service-offerings and architecture approaches
-
Updated
Jul 2, 2020
This repo is to demonstrate rag data processing pipeline using dataflow flex templates
gcp embeddings dataflow-job rag gcp-dataflow cloud-storage-bucket flex-template pickle-file genai rag-pipeline
-
Updated
Jan 2, 2025 - Python
Public GCP Data Architecture Baseline: Hybrid Warehouse/Lakehouse with Batch + Streaming
-
Updated
Mar 12, 2026 - Python
Apache beam sandbox w/ Dataflow for 10+ use cases
-
Updated
Mar 20, 2020 - Python
Big Data ETL Pipeline for ASL-to-Text (Computer Vision), using Apache Beam on GCP Dataflow
-
Updated
Feb 23, 2021 - Jupyter Notebook
Leveraged GitHub Actions to automate the deployment of a GCP pipeline for Snowflake to BigQuery data migration. Utilized 'sensex-data-analysis' as the data source and Snowflake storage integration feature to load data to GCS. Implemented workflow management and transformation using Composer (Airflow) and Dataflow
-
Updated
Feb 26, 2024 - Shell
GCP Dataflow pipeline with mapreduce in python
-
Updated
Aug 11, 2018 - Python
A data pipeline to ingest, process, store storm events datasets so we can access them through different means.
python aws-s3 apache-beam aws-athena gcp-storage apache-superset aws-glue datastudio snowflakedb sqlalchemy-python gcp-dataflow aws-quicksight gcp-bigquery
-
Updated
Apr 7, 2021 - Jupyter Notebook
An end to end anime recommendation system based on data scrapped from myanimelist.net
bigquery airflow etl anime gcp scrapy anime-recommendations anime-scraper gcp-cloud-functions gcp-pubsub kubeflow gcp-composer gcp-dataflow gcp-sql tensorflow2 gcp-kubernetes vertex-ai gcp-memorystore gcp-app-e
-
Updated
Mar 27, 2022 - Python
This project illustrates real-time data processing and analytics. This project uses Apache Kafka for capturing and streaming real-time data, GCP Cloud Functions for processing data in real-time, GCP PubSub for real-time notifications, and GCP Looker Studio for real-time data visualization.
visualization velocity value snowflake google-cloud-functions volume apache-kafka real-time-analytics real-time-processing apache-airflow gcp-pubsub gcp-dataflow looker-studio
-
Updated
Sep 10, 2024 - HCL
-
Updated
Jan 29, 2019 - Scala
Improve this page
Add a description, image, and links to the gcp-dataflow topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the gcp-dataflow topic, visit your repo's landing page and select "manage topics."