A generic ETL framework with Spark_SQL for transforming data by constructing pipelines with Yaml/Json/Xml

Simple and Distributed Machine Learning

Apache Spark Connectors for OceanBase.

A library to transform Scala product types and Schemes from different systems into other Schemes. Any implemented type automatically gets methods to convert it into the rest of the types and vice versa. E.g: a Spark Schema can be transformed into a BigQuery table.

This project helps us understand customer shopping habits by looking at e-commerce data. We use special tools to clean the data, find important information, and then save it for later use. This helps businesses make better decisions.

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

Spark Connector to read and write with Pulsar

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.

sparkmobility-scala: Scala Spark codesapce for sparkmobility

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.

A recommender system for discovering GitHub repos, built with Apache Spark

An end-to-end data engineering and analysis project to process a large-scale movie dataset, derive actionable business insights using Apache Spark, and build a content-based recommendation system.

Students Academic Performance Analysis using Apache Spark

An end-to-end batch data pipeline for e-commerce analytics, built with Scala, Spark, HDFS, Hive, Postgres, and Jenkins. The system is enhanced with a parallel streaming pipeline using Kafka for real-time Twitter trend analysis.

Scalable data engineering platform for agricultural IoT sensor data with Kafka, Spark, S3, and Grafana.

Deploy Spark course examples

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/