Spark Streaming Big Data Hadoop

Implementation of data processing in the MapReduce model.

Import and process HDF5 files on Spark with Hadoop

Distributed computational problem-solving project, which aims to perform large-scale graph matching using cloud computing technologies. The project allows users to import two directed graphs and analyze the differences between them.

Library for remote JVM ExecutorService with only dependency being password-less SSH -- Run clustered Hadoop/Spark jobs from IDE -- IDE-pimped Spark shell with full auto-completion!

Scope of this project is to calculate Daily Revenue from retail products

A system to collect, analyze and visualize the tweets

An ETL pipeline that extracts data from HDFS , transforms using spark and writes back to HDFS.

Hadoop balancer which helps balance disks on a single node

Lab with Scala and Spark Streaming

A skeleton to generate a Spark job project in Scala with local distributed environment for development, example at (https://github.com/s3ni0r/spark-app-example)

Batch ETL data pipeline built on HDP 3.0 to process daily sales and business data to procedure power Bi reports. Automated the pipelines using Airflow.

Data pipeline using nifi, kafka and spark

This is the implementation of an algorithm that finds traceability links in two graphs such that the other graph is a perturbed version of the original graph.

Spark CSV to Elasticsearch

大数据学习

hadoop, spark...

some codes are created when learning BigData