From March 14-16, ENCCS hosted a hackathon event at the RISE offices on the KTH campus in Stockholm, aimed at research software developers interested in porting or optimising their code to run on AMD Instinct™ GPUs using GPU programming frameworks such as HIP, OpenMP, OpenACC, or SYCL.
The event utilized the Dardel supercomputer, Sweden's flagship supercomputer operated by PDC at KTH in Stockholm, which is accessible for academic research through NAISS. Participating teams were granted access to the Dardel system ahead of the hackathon.
The event had an impressive turnout, and eight teams were ultimately invited to join. The interest was indeed larger than what could be accommodated based on the number of expert mentors, but ENCCS will continue organising GPU programming hackathons in the future!
The teams submitted a detailed description of their project and their code's potential impact on a specific organization or the wider community. Prior to the hackathon, accepted teams were expected to engage with recommended learning resources, profile their code, and virtually meet with their assigned mentors.
The hackathon kicked off with an online day on March 7, where teams were matched with mentors and treated to introductory seminars on GPU programming by seasoned experts from AMD and HPE. George Markomanolis, a Principal Member of Technical Staff at AMD, provided an in-depth walkthrough of compilation aspects and profiling tools like Omniperf, while John Levesque, a Senior Distinguished Technologist at HPE, offered a comprehensive introduction to the HPE programming environment and profiling tools. Johan Hellsvik, application expert at the PDC centre of HPC at KTH, also gave an introduction to the PDC environment.
During the primary in-person segment of the hackathon from March 14-16, participating teams received invaluable guidance from expert mentors representing HPE, AMD, PDC, and ENCCS. Each day commenced with a stand-up session, where teams shared their progress, challenges, and daily goals. Based on these discussions, mentors were assigned to provide appropriate support.
Although teams and mentors spent a significant portion of their day in separate rooms, concentrating on their projects, a vibrant sense of community was palpable during coffee breaks, on the hackathon chat channel, and especially during social events. In fact, teams and mentors gathered informally at a renowned Stockholm pub on the eve of the hackathon on March 13, and an official dinner took place on the evening of the 14th.Additionally, participants enjoyed an exceptional guided tour of the PDC machine room, led by Luca Manzari and Gert Svensson, the system manager and deputy director of PDC, respectively. The tour offered an in-depth look at the complexities of liquid cooling, backup power, and fire safety systems, along with intriguing stories and historical tidbits about PDC.
Overall, the hackathon proved to be a valuable opportunity for research software developers to enhance their GPU programming skills and collaborate with expert mentors in achieving their goals. The event was a resounding success, and we eagerly anticipate hosting similar events in the future!
For a more detailed account of the hackathon experience from the point of view of the project teams, please read on for stories from the UppASD and IFS teams!
UppASD
UppASD is a program for simulation of atomistic spin dynamics and spin-lattice dynamics. The program is written in Fortran 2003 with shared memory parallelization over CPU cores by means of OpenMP. Team UppASD entered the hackathon with the expectation to get expert advice on which programming model to work with for GPU offloading of compute intensive kernels, and get started with the implementation. Being agnostic at first on the choice of HIP, OpenSYCL, and OpenMP, the team settled for working with OpenMP. The first step was to obtain a better understanding of the performance of the current OpenMP parallelized CPU code. Work pursued to implement offloading directives for one of the dominant terms of the Hamiltonian, namely bilinear spin-exchange, as well as for the semi-implicit midpoint integrator. The good progress made during the hackathon forms a platform for porting all main parts of UppASD to GPU code.
IFS
Two teams from the ECMWF offices in Bonn and Reading travelled to Stockholm. In their suitcase: two components of the weather forecasting model IFS, which is currently being deployed and optimized on LUMI for the first Digital Twins of the European Commission’s Destination Earth initiative.
The objective of the first team: to make the existing initial offload implementation of the spectral transformation library, a key component and one of the most computationally expensive parts of the IFS, run faster on AMD MI250X GPUs. The second team worked on a proxy application for the physical parameterisations, with the goal to develop an optimal GPU adaptation recipe. For that, they could draw from a wide range of already available programming model implementations of the same algorithm, trying to make as many work as fast as the reference results on NVIDIA A100 GPUs.
IFS is written mostly in Fortran, and early on teething problems of the relatively juvenile software stack for AMD GPUs were encountered. But the Cray compiler could ultimately be convinced to generate working offload binaries for both applications. With the help of the fabulous mentors, profiling efforts were soon successful, and first optimization approaches identified. After three days, the spectral transforms team could enter a 4x speed-up via a mixture of targeted optimisations. The physical parameterizations application managed to achieve on-par performance with NVIDIA A100 GPUs for a HIP implementation of the algorithm but failed to achieve improvements on other programming model implementations, in particular pragma-based approaches.
Nevertheless, a positive outcome for all, who returned home with a suitcase that was filled with faster code, a lot of newly acquired knowledge, memories of positive interactions with mentors and other participants, and an even longer list of ideas and tasks they want to tackle next.
