Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

Smart Automation Tool for building modern Data Lakes and Data Pipelines

High performance data store solution

Data Lineage Tracking And Visualization Solution

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

An end-to-end batch data pipeline for e-commerce analytics, built with Scala, Spark, HDFS, Hive, Postgres, and Jenkins. The system is enhanced with a parallel streaming pipeline using Kafka for real-time Twitter trend analysis.

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

Dynamic Conformance Engine

基于Spark+SparkMLlib+Debezium+Deequ打造的简单易用、超高性能大数据治理引擎。适用于批流一体的数据集成和数据分析，支持CDC实时数据采集、机器学习算法模型、数据质量校验、数据标注、敏感数据识别、数据建模、算法建模和OLAP数据分析

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

Projects from TrendyTech

This project is a data processing application built with Apache Spark and Scala. This is designed to efficiently process, analyze and transform large datasets related to people data. It leverages Spark’s distributed computing capabilities to handle scalable data ingestion, cleaning and reporting. Shell scripts are included for hadoop deployment.

For artificial intelligence with sbt to build - maven build pls step forward to https://github.com/bgfurfeature/AI

Architecture et développement des systèmes distribuées à large echelle

全国大数据竞赛三等奖解决方案，省赛二等奖解决方案。一键安装大数据环境脚本，自动部署集群环境，包括zookeeper、hadoop、mysql、hive、spark以及一些基础环境。已通过实际服务器测试，效果极佳，仅需要输入密码等少量人为干预。解放安装部署配置所需人力。并添加若干scala案例，结合spark用以进行数据准备。

Batch process that compacts different parquet files stored at Azure Data Lake Storage following the requirements specified at README.

An application for DNA sequence analysis written in Scala and deployed using Apache Spark&Hadoop.

hadoop, spark...

深圳地铁大数据客流分析系统🚇🚄🌟