The Language Technology Group (LTG) at the University of Oslo has prepared three Norwegian large language models (LLMs). It became possible as researchers were given time to access Europe’s most powerful computer, Lumi, situated in Finland. Researchers from LTG trained their large language models (LLMs) on Lumi before Christmas and processed quite a large chunk of data to launch three different Norwegian language models to challenge ChatGPT. Access to Lumi is not easy to get, as researchers have to apply in advance and then wait for their turn, but the powerful computer does speed up the process of data training.
Why Norwegians want their own large language model (LLM)
Norwegian wanted to build their own language model, as there were many uses beside being in the native language for broader adoptability. Pointing to different issues associated with commercial solutions, Oslo University’s informatics professor, Erik Velldal, said,
“There are many problems associated with the tech giants’ language models. They appear as black boxes to the outside world. We need Norwegian alternatives.”
Source: Erik Velldal.
Earlier, the National Library of Norway and the University of Oslo worked on a handful of Norwegian language models, but the ones developed now are the largest so far, with more than 30 billion words or training.
Lumi’s role is important as a lot of computing power is required for training AI models, including many GPU units. The training process can be sped up if the number of GPUs is increased, and Lumi offers a good opportunity as it has more than 10,000 GPUs incorporated into it.
Associate Professor Andrey Kutuzov says that among many other reasons for a Norwegian language model, were that the likes of ChatGPT don’t adapt well to the knowledge and values of Norway. These models are trained on American languages and English mainly, so definitely they reflect American culture and values as well, and what Norwegian people needed was something according to their own preferences. He says,
“A Norwegian language model will to a much greater extent, reflect society as we know it in Norway.”
Source: Andrey Kutuzov.
Limitations of Norwegian large language models
The Norwegian models, essentially intended for developers and researchers, have already been launched and have been downloaded by several thousand people. Kutuzov explains that these models are not available online on the web. As the web interface is considered easy to use by the general public. According to the professor, these models are still far from commercial models in terms of the possibilities they offer. As these are general base models,.
It is a perception in the Norwegian language tech group that they must have their own models to compete with OpenAI’s ChatGPT or Google’s LaMDA. Along with many other observations they had, like the commercial models only give access to them through their web interface and the model behind the system is closed for access. So in many contexts, it can be risky to give out data to a third party service provider. Like medical data, where privacy is the biggest concern.
Norwegian language models are quite limited as compared to ChatGPT. As they are trained on only 0.1% of the data training done for ChatGPT, but these models are not closed and not restricted, so they can better serve researchers.
Matching the qualities and standards of American language models is still a tough task, and Norwegian models need further development. To be on par with ChatGPT or Bard, they need instructional training so that they can solve a bit more complex problems. It is said that new versions of the models will be launched, and the process will continue.
Find more information at the University of Oslo website.
