Posts by Mehdi Rezagholizadeh
Pre-Training Hybrid LLMs from Scratch in Primus on AMD Instinct™ GPUs
- 08 October 2026
In this blog you will learn how to pre-train hybrid LLMs from scratch with Primus on AMD Instinct™ MI355X GPUs. In particular, we will cover how to declare a hybrid layer stack with a single configuration string, how three linear-recurrent mixers (Mamba2, Gated DeltaNet, and Kimi Delta Attention) trade off throughput with training quality, which recipe settings determine whether your run fits in memory in the first place, and which environment variables can quietly change your results. Along the way you will receive the released configurations behind every number in this post, so that you can reproduce the runs yourself. Ready to train your own hybrid? Pull the Primus image and follow along — everything here runs on a single eight-GPU node.
Zebra-HyLo: Upcycling Transformers into Long-Context Hybrid LLMs on AMD Instinct™ GPUs
- 22 September 2026
Hybrid language models that interleave attention with linear sequence-modeling blocks have become the default answer to the cost of long context. Jamba, Samba, Qwen3-Next and Kimi-Linear all take this shape, and they all share one property that makes them expensive to adopt: they are pretrained from scratch. Every one of them pays the full cost of building a foundation model again, which means the enormous investment already sunk into existing Transformer checkpoints is thrown away.
AgentKernelArena: Benchmarking AI Coding Agents for GPU Kernel Optimization on AMD Instinct GPUs
- 03 July 2026
AI coding agents such as Cursor Agent, Claude Code, and OpenAI Codex are improving fast, and people increasingly trust them with specialized, high-stakes work, including GPU kernel optimization. But most of the public evidence is still a cherry-picked demo on a single kernel, not a controlled, head-to-head comparison on the same tasks, the same hardware, and the same scoring rules. On AMD Instinct™ GPUs, where every percentage point of kernel performance translates directly into training and inference cost, that gap matters.
AMD-HybridLM: Towards Extremely Efficient Hybrid Language Models
- 17 September 2025
The rapid rise of deep learning applications has intensified the demand for language models that offer a balance between accuracy and efficiency—especially in settings constrained by memory, compute, or real-time requirements. While Transformer-based models have revolutionized natural language processing, their quadratic attention complexity and large key–value (KV) cache requirements pose serious challenges for deployment, particularly on edge devices or in latency-sensitive environments.