AI consultancy company 42 Labs accesses EuroHPC JU supercomputer MareNostrum 5 to train a Swedish large language model (LLM). They applied using the Benchmark Access Call which is continuously open all year round.

Organization Involved 

42 Labs is a technology consultancy company that specialises in developing customised conversational AI solutions. The company has strong expertise from both big tech companies and startups in building LLMs tailored for specific data and business needs. They specialise in building advanced RAG-based systems, knowledge base extraction from documents and integration with systems of record and engagement, using both cloud and on-prem deployments.  

Technical/Scientific Challenge 

42 Labs has developed a small Swedish LLM called Lynx Micro 2B, a Gemma 2B-refinement based on high-quality single-turn and multi-turn Swedish instruct data. While quite small (2B parameters), the model was able to outperform most models on Scandeval (excluding OpenAI models). The aim is now to train a larger, more capable version of this model. 

Proposed Solution 

Using open-source tools from Huggingface, 42 Labs will fine-tune a larger model aiming at 7-9B parameters starting from Llama/Gemma. Fine-tuning is distributed over several NVIDIA H100 GPUs, using the flash-attention algorithm to improve memory access performance. Then, the scandeval package is used to evaluate model performance. 

Business Impact 

Benchmark access to MareNostrum 5 has been instrumental in being able to exploit high-performing GPUs (NVIDIA H100) in a cost-effective manner. Fine-tuning a Llama 3.2 3B model gave good results and successful upscaling of the training can lead to an improved open-weight model.

Benefits 

• The HPC allocation allowed to scale fine-tuning to dozens of GPUs at no cost 
• The successful tests on the Llama 3B model give promising results for training scale-up 
• Insights from this work will be very useful in upcoming projects, where a larger-scale training is envisioned. 

Do you need more computational power for your projects?

Learn more about how you can access supercomputers for your projects. ENCCS can help! Visit https://enccs.se/get-supercomputer-access/