This repository is about data processing using spark and flink

A Rule-Based discount calculating engine for a retail store. The engine reads transaction data, applies various discount rules based on the product, date, and other criteria, and writes the results back to a database.

Data Brewery is an ETL (Extract-Transform-Load) program that connect to many data sources (cloud services, databases, ...) and manage data warehouse workflow.

Extract, Transform, and Load pipeline

生产环境的海量数据计算产品，文档地址：

A simple ETL pipeline in Scala utilizing the Typelevel libraries.

Let's be honest - most data pipeline frameworks treat types as suggestions. Config files are strings. Schemas are "validated" at runtime. Data quality is an afterthought. So, let's do differently

data pipeline using the lambda architecture is created for the unification and consolidation of real-time customer web events, weblogs, and profile data into a hive warehouse for adhoc analysis.

Sample notebooks on Azure Databricks for ETL

Simple Spark project built in Scala

A project to use spark for transform the *.n3 file to CSV file

Repository for playing with spark

Playing around with Apache Spark.

This Scala ETL pipeline processes transaction CSV files by applying discounts and calculating final prices using pure functional programming principles. The transformed data is then loaded into an Oracle database, demonstrating end-to-end data processing with type safety and immutability.

This project is an example for reading data from Impala (using impala for transformation) as a Spark DataFrame and writing objects from Spark into oracle database using JDBC.

Cross database data transfer tool

Dataport Pecan Street energy profile extractor.

Extend JDBC types support for Apache Spark.