EuroHPC systems have different types of storage that cater to different needs. Generally speaking, three types of storage are commonly available across HPC clusters:

• Home storage, where config files, caches and personal data of each user are saved. It is usually backed up and is relatively small in size;
• Project storage, which is usually shared among all project members. It is used to store applications and/or data that needs to be available throughout the duration of the project;
• Scratch space for files and data produced during job execution;
• Some variation of faster (but smaller) storage for when I/O is a bottleneck.

The first two are shared across all nodes using a parallel file system solution (usually Lustre), while the latter tends to be node-local (or directly in RAM). Moreover, sometimes a long-term storage option is present. Taking as example the three flagship EuroHPC systems, we will look at how to access the different types of storage and how to use them effectively.

LUMI

On LUMI, the home folder can occupy up to 20 GB. Each project allocation is identified by project_XXXXXXXXX, where XXXXXXXXX is a 9-digit number. Each allocation is awarded the following storage:

• Project application space under /projappl/project_XXXXXXXXX (also symlinked to /project/project_XXXXXXXXX). It has a size of 50 GB and it should be used to store applications and code to be used by all project members;
• Project data (also called “scratch”) stored under /scratch/project_XXXXXXXXX. Its base size is 50 TB and it is where data needed throughout the lifetime of the project should be stored;
• Fast storage under /flash/project_XXXXXXXXX. Its size is 2 TB and, while not being node-local, it is faster than project storage.

Moreover, the /tmp directory is allocated directly in RAM and there are long-term storage alternatives (LUMI-O object storage).
The lumi-quota and lumi-workspaces commands can be used to check storage usage.

Leonardo

On Leonardo, it is not necessary to learn the actual paths of storage directories, which are instead managed through env variables:

• $HOME has a capacity of 50 GB;
• $WORK is the main project storage, it has a capacity of 1 TB per project (extendable) and data is preserved throughout the duration of the project, although it is not backed up;
• $CINECA_SCRATCH is user-specific and is temporary HDD-based storage for job runtime data and files. By default, no quota is enforced, but files are cleaned after 40 days;
• $FAST is project-wide and based on NVMe SSD drives; it has a capacity of 1 TB and is used when fast access to data is needed for all members of a project;
• $TMPDIR is node-local and is either on local SSD disks (DCGP nodes) or in RAM (Booster GPU nodes).

The cindata and cinQuota commands can be used to inspect storage usage.

MareNostrum 5

Storage on MN5 is based on the IBM General Parallel File System (GPFS), a similar solution to Lustre.

• The home directory is hosted under /gpfs/home/<username>;
• Project-wide storage is available under /gpfs/projects/<PROJECT_NUMBER>. Its size depends on approval during the application process;
• /gpfs/scratch/<PROJECT_NUMBER> can be used to store data produced during job runs;
• Node-local storage can be used during job execution under /scratch/tmp/$JOBID. This directory is cleaned after the job finishes.

Usage quotas can be inspected with the bsc_quota command.

Pro-tip: An easier way to work with all these storage systems is to export environment variables similar to how Leonardo does it. Assuming that a folder with your username was created in the project storage, the following lines could be added to the .bashrc file in your home folder (e.g. for MN5)

export WORK=/gpfs/projects/<PROJECT_NUMBER>/<username> 
export SCRATCH=/gpfs/scratch/<PROJECT_NUMBER>/<username> 
export FAST=/scratch/tmp/$JOBID # This is also called $TMPDIR on MN5

This way, these variables can be used in any shell (and job script) abstracting away the difference between the different clusters.

More ENCCS tutorials

Check more of our ENCCS short tutorials here.