Generative AI is currently the hottest ticket in tech, with major players like Microsoft, Google, Amazon, and Meta dominating the scene with their large language models (LLMs). These LLMs are widely seen as the next major breakthrough in technology. However, there’s a glimmer of hope for Europe as it seeks to challenge Big Tech’s supremacy, especially in languages beyond English.
The compute and data dilemma
Big Tech’s dominance in the AI arena is driven not only by their vast computing power but also by access to extensive language data, often scraped from the internet. While it may be a tall order for a European company to outperform Big Tech in building an English LLM, a Finnish AI company named Silo has released promising results for its multilingual model called Poro (Finnish for reindeer).
Poro is trained on both Finnish and English text, serving as proof that high-performance LLMs can be created by blending different languages. Silo claims that early results demonstrate its competitiveness with Meta’s open-source Llama models. Silo’s collaboration with the University of Turku has leveraged a treasure trove of data from the EU-funded initiative known as The High Performance Language Technologies (HPLT) project, which has collected a staggering 7 petabytes (7,000 terabytes) of language data across 80 languages since 2022. To put this in perspective, GPT-3.5, which powered ChatGPT’s release version, was trained on 45 terabytes of text data.
Access to such high-quality, publicly-funded text data, as in the case of HPLT, could give models like Silo’s an edge in languages with limited online data availability.
The multilingual approach
Due to the scarcity of data for languages like Finnish, Silo adopted a multilingual strategy by “cross-training” its model with both English and Finnish data. This approach involves feeding the model text in both languages, allowing it to learn how the two languages relate to each other. As a result, the model can generate responses in Finnish even if it hasn’t encountered Finnish code before.
Peter Sarlin, Silo’s co-founder and CEO, explains, “You’re able to generate code in Finnish even though the model has not seen any Finnish code.” Silo plans to open-source its cross-training techniques, potentially paving the way for the development of models across all European languages, including those with limited available data.
The quest for technological sovereignty
Sarlin points out that there is a significant opportunity in the market for LLMs in languages other than English. He emphasizes the importance of European businesses not relying solely on technology owned by large U.S. companies. Building on Big Tech AI models might result in little value creation staying in Europe. Therefore, there’s a growing imperative for Europe to assert its technological sovereignty.
Supercomputing power from LUMI
Poro’s training also benefited from the EU-funded supercomputer known as LUMI, which became operational in 2022. Notably, LUMI uses AMD chips, a departure from the industry-standard NVIDIA chips. While some consider AMD chips to be expensive and inefficient for AI, Silo’s team invested significant resources in developing AI training software optimized for them.
Sarlin indicates that they plan to open-source a substantial portion of this software and are committed to assisting other companies in training models on LUMI. If European companies can harness resources like LUMI for AI training, it could be a game-changer in the continent’s quest to assert itself in the age of AI.
The European renaissance in AI
As Big Tech continues to flex its AI muscles in Silicon Valley, Europe is quietly building its own arsenal. The continent’s strengths lie in its multilingual approach, leveraging publicly-funded language data, and exploring alternative computing options like LUMI. While the road ahead may be challenging, Europe’s determination to compete on the global AI stage is evident.
Europe’s secret weapon in the battle against Big Tech’s dominance in generative AI is rooted in its diversity, collaboration, and commitment to technological sovereignty. With Silo’s Poro model as a testament to what can be achieved, Europe may well have a fighting chance to leave its mark in the world of AI.
