Posts by Pavel Emeliyanenko
Group-GEMM in XLA: Making Mixture-of-Experts Fast and Memory-Efficient
- 26 October 2026
Mixture-of-Experts (MoE) models have become a standard in most recent LLMs (DeepSeek, Grok-1, Llama-4, Gemini). But under the hood, an MoE layer turns into something that GPUs and math libraries have historically been bad at: a large collection of small, independent matrix multiplications, each with a different, data-dependent shape.