Innovations in Generative AI Hardware are making strides toward affordability, accessibility, and efficiency, challenging the constraints posed by the exponential growth in the size of large language models (LLMs). In a recent panel discussion, industry leaders shared insights into their strategies for addressing these pressing challenges.
Marshall Choy, Senior VP of Products at SambaNova Systems, highlighted the critical role of memory architecture in bringing down the cost of using LLMs. As LLMs boast parameter counts reaching the billions or trillions, the focus has shifted towards memory as the bottleneck. SambaNova Systems has adopted a three-tier memory architecture, addressing latency, bandwidth, and capacity within a single framework. This innovative approach aims to economically scale the usage of LLMs, where memory efficiency is the linchpin.
Democratizing large models
The ballooning size of LLMs presents a significant accessibility challenge. Once a model exceeds a trillion parameters, the associated hardware and operating costs become prohibitive, leaving their utilization confined to a select few. To make large models accessible to a wider audience, SambaNova Systems has introduced a novel concept known as “composition-of-experts.”
This approach diverges from the conventional “mixture-of-experts” paradigm, where complex predictive modeling problems are divided into subtasks. Instead, SambaNova trains domain expert models for precision and task relevance, assembling a trillion-parameter composition-of-experts model. This model can be continually trained on new data without sacrificing prior learning, all while minimizing compute latency and reducing the costs associated with training, fine-tuning, and inferencing.
Efficiency through model techniques
Efficiency in generative AI hardware goes beyond the hardware itself; it extends to the relationship between model architecture and the hardware it runs on. Matt Mattina, VP of AI Hardware and Models at Tenstorrent, emphasized the importance of breaking the feedback loop where model architecture is shaped by the hardware it’s trained on.
Tenstorrent adopts techniques like network architecture search with hardware in the loop, allowing trainers to specify the hardware for inference during training. This paradigm shift ensures that models are tailored not for the training machine but for the ultimate inference machine, leading to more efficient models for practical use.
Specialization at the system level
AI is an ever-evolving field, posing challenges in balancing dedicated chips and custom silicon with system flexibility. Jeff Wittich, Chief Product Officer at Ampere Computing, offers a perspective that favors specialization at the system level. He asserts that this approach provides the flexibility to mix and match components, creating versatile solutions capable of adapting to the rapidly changing AI hardware landscape.
Traditionally, creating and commercializing new hardware takes several years. Ampere’s partnership with companies developing various training and inference accelerators aims to achieve the right balance. Ampere envisions improved performance and efficiency by coupling general-purpose CPUs with accelerators specialized in specific tasks.
Integration benefits and flexibility
Wittich highlights the importance of integration, which should ideally enhance performance and efficiency without sacrificing flexibility. The fusion of general-purpose CPUs with specialized accelerators is seen as a promising avenue. Over time, the tight integration of these accelerators with CPUs is expected to further optimize AI workloads. The key principle remains: integration should enhance capabilities without imposing restrictions.
