Multi-GPU Programming Bootcamp
May 6 • 09:00 - May 7 • 14:30 CEST
Overview
Welcome to the Multi-GPU programming bootcamp!
Scaling applications to multiple GPUs across multiple nodes requires one to be adept at programming models and optimisation techniques, and proficient at performing root-cause analysis using in-depth profiling to identify and minimise bottlenecks.
Topics on GPU programming covers single node multi-GPU (P2P) and multi-Node multi-GPU (GPU Direct) programming, the NVIDIA Collectives Communications Library (NCCL), and the NVSHMEM, which is a parallel programming interface based on OpenSHMEM that provides efficient and scalable communication for NVIDIA GPU clusters).
The Multi-GPU Programming Bootcamp covers step-by-step ways to improve application performance using cues from profilers and provides an understanding of the underlying technologies and communication topology needed to leverage powerful NVIDIA® libraries to extract more performance from the system.
Prerequisites
- Experience with C/C++, NVIDIA CUDA®, MPI
Agenda
Friday, May 3, 2024
| Time (CET) | Topic |
|---|---|
| 11:00–12:00 | Cluster Dry Run Session |
Monday, May 6, 2024
| 09:00-09:05 | Welcome |
|---|---|
| 09:05-09:30 | Introduction to Nsight Systems and GPU Computing (Lecture and Read-only Lab) |
| 09:30 -10:30 | Overview of a Single GPU Code |
| 10:30 -12:30 | Parallelizing a Single GPU Code using CUDA memcpy, Streams, and an Introduction to Single Node Multi-GPU ( P2P ) (Lecture and Lab) |
| 12:30-13:00 | Lunch |
| 13:00-15:00 | Introduction to Multi-Node Multi-GPU (GPU Direct ) programming (Lecture and Lab) [2 hrs] |
| 15:00-16:00 | LLM Projects Discussion (Optional) |
Tuesday, May 7, 2024
| Time (CET) | Topic |
|---|---|
| 09:00-11:00 | Introduction to NVIDIA Collectives Communications Library (NCCL) (Lecture and Lab) |
| 11:00-13:00 | Introduction to NVSHMEM (Lecture and Lab) |
| 13:00-13:30 | Demo of the Nsight Systems Profiler Report for a Multi-Node Application |
| 13:30-14:30 | LLM Projects Discussion (Optional) |
Compute Resource
Attendees will have the opportunity to use a GPU cluster for the duration of the bootcamp.
Registration
Please register using this link https://www.openhackathons.org/s/siteevent/a0C5e000008AbEfEAK/se000316
Disclaimer
Due to EuroCC2 regulations, we cannot except generic or private email addresses. Please use your official university or company email address.
This training is intended for users established in the European Union or a country associated with Horizon 2020. You can read more about the countries associated with Horizon2020 here https://ec.europa.eu/info/research-and-innovation/statistics/framework-programme-facts-and-figures/horizon-2020-country-profiles_e
Organisers
This Bootcamp is co-organized by ENCCS, EuroCC Austria, EuroCC Germany in association with (HLRS), Jülich Supercomputing Centre (JSC), Leibniz Supercomputing Centre (LRZ), Vienna Scientific Cluster (VSC), OpenACC organization, and NVIDIA.
ENCCS Lesson Materials
Have a look at ENCCS lesson materials where you can find multiple lessons on GPU programming, data analysis and HPC optimisation.
Related Events
This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 951732. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and its associated countries .