Hadoop MapReduce Programs using Scala to process log files.

The project developed for the second module of the course "Languages and Algorithms for Artificial Intelligence", focused on Functional Programming with Scala and Spark.

Parsing the common crawl database using Scala and Spark

Group 10 Project, Fall 2020, CS 6240: Large-Scale Parallel Data Processing, Khoury College of Computer Sciences, Northeastern University

A simple spark job to test deployment, oozie and emr configurations.

This repository contains the final project for the course Big Data Systems Engineering with Scala.

Infrastructure: The projects herein simplify the repeated use of a variety of frameworks, and cloud services & platforms.

Offline Elasticsearch index generator

Continuing with my case study on reading a big data file, this is the fourth part in my increasingly innacurately named trilogy on how I got on reading a bigg'ish file with C, Python, spark-python and spark-scala.

Automated tool for choosing which Wikipedia page to translate to a dedicated language, built for EMR from AWS

Half-baked implementation of a cluster manager for EMR.

Spark code used for my Master's Thesis. Run on AWS EMR clusters

Link Prediction is about predicting the future connections in a graph. In this project, Link Prediction is about predicting whether two authors will be collaborating for their future paper or not given the graph of authors who collaborated for atleast one paper together.

Hadoop Map Reduce

Collection of code for submitting Spark/Hadoop/Hive/Pig tasks to EMR (AWS Elastic MapReduce) | #DE

A boilerplate for spark projects with docker support for local development and scripts for emr support.

SBT plugin for Apache Spark on AWS EMR