Faculty of Computer Engineering Informatics
and Communications
Department of Computer Engineering
Module: Parallel and Distributed Computing
Module Code: HCE/HHE/HSE 311
Lecturers: Mr. I Chikukuza/Eng. A. Mukwazvure
Email Address: ichikukuza@ceic.uz.ac.zw/amukwazvure@ceic.uz.ac.zw
Mobile No: 0719311812/0777740504
Objectives:
This course will provide students with practical and theoretical knowledge
Parallel and Distributed Computing. On completion of this course, students
should be able to:
Students should gain an understanding of the principles, concepts, and
challenges of distributed computing.
Students should learn about parallel processing architectures, parallel
algorithms, and techniques for exploiting parallelism in computation.
Students should learn techniques for optimizing the performance of
distributed and parallel applications.
Students should understand the challenges of fault tolerance in
distributed and parallel systems.
Students should understand the concepts of scalability and resource
management in distributed and parallel computing.
Students should stay updated with the latest trends and technologies in
distributed and parallel computing, such as cloud computing, edge
computing, and big data processing frameworks.
Learning Outcome:
At the end of the course, the students will be able to do following:
Students should acquire a solid understanding of the principles,
concepts, and theories related to distributed and parallel computing.
Students should be able to analyze complex computational problems
and determine whether they can benefit from distributed and parallel
computing approaches.
Students should learn techniques for optimizing the performance of
distributed and parallel applications.
Students should develop skills in collaborating with others,
understanding and respecting diverse perspectives, and effectively
communicating and coordinating tasks in distributed and parallel
computing projects.
Students should develop critical thinking skills to evaluate the
advantages, limitations, and trade-offs of different distributed and
parallel computing approaches.
Students should be aware of the ethical implications and considerations
associated with distributed and parallel computing
Students should develop a mindset of continuous learning and stay
updated with the latest advancements and technologies in distributed
and parallel computing.
Topic Content
Introduction to Parallel and Overview of parallel and distributed computing
Distributed Computing concepts
Parallel and distributed computing strategies
Benefits of parallel and distributed computing
Fundamental differences between parallel and
distributed computing
Examples of real-world applications
Parallel Computing Architectures Flynn's taxonomy: SISD, SIMD, MISD, MIMD
Shared memory architectures (e.g., multicore
processors)
Distributed memory architectures (e.g.,
computer clusters)
Hybrid architectures (e.g., GPUs)
Hardware considerations in parallel computing.
Scalability and performance considerations
Models of Computation Overview of parallel computation models
(PRAM, BSP).
Distributed system models (client-server, peer-
to-peer).
Characteristics of parallelism: data, task, and
pipeline.
Distributed Systems Characteristics and challenges of distributed
Fundamentals systems.
Communication models: message passing vs.
shared memory.
Synchronization and coordination in distributed
environments.
Parallel Algorithm Design Design strategies: divide and conquer, dynamic
programming.
Case studies of classic parallel algorithms (e.g.,
parallel sorting, matrix operations).
Complexity analysis and performance
evaluation.
Programming Models and Tools Overview of programming models (OpenMP,
MPI, CUDA).
Introduction to high-level frameworks (Apache
Spark, Hadoop).
Practical exercises in parallel programming.
Distributed Algorithms Consensus and coordination algorithms (Paxos,
Raft).
Distributed mutual exclusion and leader
election.
Analysis of distributed algorithms’
performance.
Distributed Computing Definition of terms
Environments, Coordination and Key Characteristics of Distributed Computing
Synchronization Environments
Types of Distributed Computing Environments
Key Components of a Distributed Computing
Environments
Challenges of a Distributed Computing
Environments
Coordination Techniques
Synchronization Techniques
Performance Evaluation and Metrics for evaluating performance (speedup,
Scalability efficiency).
Amdahl’s and Gustafson’s Laws.
Identifying and addressing performance
bottlenecks.
Fault Tolerance and Reliability Techniques for fault detection and recovery.
Replication and consistency models in
distributed systems.
Case studies on fault tolerance strategies.
High-Performance Computing What is High-Performance Computing (HPC)
Key Features of HPC
Applications of HPC
Challenges in HPC
Supercomputer What is a supercomputer
Key Features
Applications
Notable Examples
Challenges
High Performance Cluster What it is High Performance Cluster
Key Features
Applications
Advantages
Challenges
Quantum Computing Quantum Computing
Key Concepts
Applications
Advantages
Challenges
Distributed Computing What is big data analytics
Applications in Big Data Analytics Key Concepts
Applications in Big Data Analytics
Advantages
Challenges
Emerging Trends in Parallel and Cloud computing and virtualization
Distributed Computing Edge and fog computing
Internet of Things (IoT) and sensor networks
High-performance computing (HPC) clusters
and supercomputers
Quantum computing and its implications
Reference Books
Distributed Systems: Concepts and Design" by George Coulouris, Jean Dollimore,
and Tim Kindberg (2011)
Parallel Computing: Theory and Practice" by Michael J. Quinn (2018)
High Performance Computing" by Charles Severance and Kevin Dowd (2018)
Big Data: Principles and best practices of scalable real-time data systems" by
Nathan Marz and James Warren (2015)
Assessment Methods
Coursework
Theory Assignments
Presentations
In-class tests
