Posts by Yonatan Dukler
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
- 24 July 2026
AMD is excited to introduce Instella-MoE, a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs with the AMD ROCm™ software stack, Instella-MoE combines a sparsely activated MoE design with architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity. Instella-MoE delivers competitive performance across a broad suite of benchmarks against both dense and MoE baselines (as shown in Figure 1) including models with comparable or larger active parameter counts, establishing it as one of the strongest fully open language models at its scale.
Accelerating Mixture-of-Experts Execution with FarSkip-Collective Models
- 05 May 2026
Whether you are running training or inference, the largest Mixture-of-Experts (MoE) based LLMs cannot fit on a single GPU; instead you must run collective-communication operations to integrate the work of multiple GPUs to work together on a single model.