Posts by Xiaoming Peng
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
- 03 September 2026
DeepSeek-AI released the DeepSeek-V4 series on April 24, 2026: a preview pair of MIT-licensed Mixture-of-Experts models, with DeepSeek-V4-Flash at 284B total parameters (13B activated) and a one-million-token context window. Flash pushes sparse attention further than any open-weight model before it: its 43 transformer layers interleave three different attention types, each layer sits inside a hyper-connection block rather than a plain residual, and every token is routed through 256 experts. Each of those choices breaks an assumption baked into stock Megatron-LM training code.
Primus-Pipeline: A More Flexible and Scalable Pipeline Parallelism Implementation
- 23 February 2026
Error parsing meta tag attribute “keywords”: No content.
MoE Training Best Practices on AMD GPUs
- 16 December 2025
This blog covers best practices for training Mixture-of-Experts (MoE) models on AMD Instinct™ MI300/MI355-series[a] GPUs with the ROCm ecosystem. Whether you’re new to MoE distributed architectures or optimizing trillion-parameter models, this guide will help you identify bottlenecks and maximize efficiency on AMD hardware.
Primus: A Lightweight, Unified Training Framework for Large Models on AMD GPUs
- 22 August 2025
Training large language models (LLMs) at scale is inherently complex. Different frameworks expose inconsistent interfaces, multi-GPU and distributed setups require brittle scripting, and backend-specific quirks introduce overhead that slows down training iterations. Primus tackles these challenges with a streamlined, backend-agnostic training framework that helps developers launch, customize, and scale training jobs faster on AMD GPUs.