Posts by Fuwei Yang
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
- 03 September 2026
DeepSeek-AI released the DeepSeek-V4 series on April 24, 2026: a preview pair of MIT-licensed Mixture-of-Experts models, with DeepSeek-V4-Flash at 284B total parameters (13B activated) and a one-million-token context window. Flash pushes sparse attention further than any open-weight model before it: its 43 transformer layers interleave three different attention types, each layer sits inside a hyper-connection block rather than a plain residual, and every token is routed through 256 experts. Each of those choices breaks an assumption baked into stock Megatron-LM training code.
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
- 18 August 2026
Reinforcement learning (RL) for large language models (LLMs) alternates between two phases: generation (rollout), where the current policy produces responses, and training, where those responses are used to update the policy. In verl, the key design choices are when these phases run relative to each other (synchronously or with overlap) and where they run (colocated on the same GPUs or on separate GPU pools). This blog first explains the differences between the two modes and when to use each.
GEAK V3: Agent-Driven, Repository-Level GPU Kernel Optimization across HIP, Triton, and FlyDSL on AMD GPUs
- 20 July 2026
In the ever-evolving world of GPU computing, optimizing kernels for performance and efficiency is a critical challenge. Hand-tuning kernels demands deep technical expertise and manual iteration. In this blog, you will read about how GEAK v3, the latest iteration of the agent-driven framework, tackles this problem using enhanced features such as task planning, test-harness discovery, patch-based handling of multi-file kernels, dynamic memory system and expert knowledge database. Our results show improvements across three kernel languages (HIP, Triton, and FlyDSL) and both CDNA and RDNA GPUs.
Triton-Based Optimization of Video Sparse Attention on ROCm
- 13 July 2026
Video generation has become a major frontier in generative modeling, driven by large-scale data and increasingly scalable architectures. Among modern architectures, Diffusion Transformers (DiTs) have emerged as a dominant paradigm [1,2,3,4] by representing videos as spatio-temporal token sequences, enabling long-range interactions across frames and spatial regions, as well as flexible multimodal conditioning with text or audio. However, full self-attention scales quadratically with token count, making it increasingly expensive as spatio-temporal resolution and model size grow. Video sparse attention (VSA) [5,6] mitigates this cost by approximating full attention with a subset of informative token interactions, but its practical efficiency in both training and inference depends heavily on hardware-aware Triton kernel implementations.
Out-of-the-Box ROLL Support on AMD GPUs: Accelerating Reinforcement Learning at Scale
- 01 June 2026
Reinforcement learning (RL) is rapidly becoming a foundational technology for Large Language Models (LLMs)—powering key abilities such as reasoning and agentic behaviors. As RL workloads grow more complex and computationally intensive, the ecosystem increasingly depends on scalable, high-performance frameworks that can fully utilize modern GPU clusters.
Accelerating Autonomous Driving Model Training on AMD ROCm™ Software
- 08 December 2025
The autonomous driving industry is undergoing rapid growth, driven by advances in AI and the increasing demand for safer, smarter transportation. At the core of this transformation are complex deep learning models that enable vehicles to perceive, reason, and navigate dynamic environments. Efficiently training these models at scale has become a key challenge for companies working to bring autonomous systems to the road.