This repository contains my solutions for the Data-Intensive Programming course completed at Tampere University.
The course focuses on large-scale distributed data processing using Apache Spark, with hands-on implementation using Python (PySpark).
Modern data systems must handle massive datasets that exceed the capabilities of traditional single-machine processing. This course explores scalable approaches to big data processing using distributed computing principles.
The implementations in this repository demonstrate:
- Distributed data processing with Apache Spark
- RDD and DataFrame transformations
- Functional-style data workflows
- Efficient large-scale computation
- Spark SQL usage (within API constraints)
- Performance-aware data engineering practices
- Apache Spark 3.5
- PySpark
- Python 3.10
- Distributed computing concepts
- Functional programming patterns
- Weekly Exercises – Incremental tasks covering Spark fundamentals and distributed processing concepts.
- Group Assignment – A larger applied project demonstrating end-to-end data-intensive workflow design.
- Resilient Distributed Datasets (RDDs)
- Lazy evaluation & execution planning
- Transformations vs. Actions
- DataFrames & Spark SQL API
- Partitioning and distributed computation
- Aggregations and large-scale joins
- Functional data transformations (map, reduce, filter, flatMap)
- Performance considerations in distributed systems
Through this course, I strengthened my ability to:
- Design scalable data pipelines
- Write distributed data processing code
- Apply functional programming in data engineering
- Understand trade-offs between performance and abstraction
- Work with big data tools in cloud environments
This repository contains only my personal implementations of coursework tasks.
No confidential course materials or restricted content are included.