RISE NLU Group will train English BERT model using multiple GPUs on the EuroHPC JU system Vega.

With the help of ENCCS, Evangelia Gogoulou from RISE NLU Group, Digital Systems got access to the EuroHPC JU system Vega at Izum, Slovenia. She will be training from scratch an English BERT model using multiple GPUs and then evaluate its downstream performance on the GLUE benchmark (https://gluebenchmark.com/ ).

The model performance will be compared with the English BERT trained on one GPU. The second task is to start from the English BERT model and continue pre-training it on Russian, on multiple GPUs. The transferred Russian model will be then evaluated on GLUE and the results will be compared with existing Russian Language models.