Indian Institute of Information Technology, Allahabad

Department of Information Technology

Course Syllabus

  1. Name of the Course:  Parallel and Distributed Computing
  2. LTP structure of the Course:  2-1-1
  3. Objective of the Course: To introduce various Parallel and Distributed hardware architectures and programming models.
  4. Outcome of the Course: Upon completion, students will be able to

a) Understand the basics of various parallel and distributed computing platforms

b) Identify the models and frameworks best suited to various workloads.

b) Provide solutions to parallel and distributed computing problems.

  1. Course Plan:

Unit

Topics for Coverage  

Unit 1

Introduction to PDC:  Latency vs. Bandwidth, Applications and Challenges, Types of architecture, Flynn’s taxonomy, Basic concepts: cores, nodes, threads, processes, speedup, efficiency, overhead, strong and weak scaling (Amdahl’s law, Gustafson’s law), Cache, Principle of Locality, Programming Models.

Unit 2

Distributed Computing: Distributed Memory, Message Passing Interface, Asynchronous/Synchronous computation/communication, concurrency control, fault tolerance, Distributed Programming with OpenMPI.

Unit 3

Parallel Computing:  shared memory, data and task parallelism, Synchronization, Concurrent Data Structures, Shared Memory Programming with available APIs: PThreads, OpenMP, TBB.

Unit 4

GPU Programming:  GPU Architecture, Programming Models:  CUDA/OpenCL, Basic Concepts:  Threads, Blocks, Grids, GPU memory hierarchy, Thread Scheduling, Warps and Control divergence, Memory Coalescing, Programming with CUDA, Using CUDA Libraries: CuBLAS, CuFFT.

  1. Text Books/References
  1. The Art of Multiprocessor Programming by Maurice Herlihy and NirShavit, Morgan Kaufmann Publishers.
  2. Principles of Parallel Programming, by Calvin Lin, Larry Snyder, Addison-Wesley.
  3. Introduction to Parallel Computing, by AnanthGrama, Anshul Gupta, George Karypis, Vipin Kumar, Second Edition.
  4. Wen-Mei W Hwu, David B Kirk, Programming Massively Parallel Processors A Hands-on Approach, Morgann Kaufmann, 3e.