Posts by Janet Tseng
Long Context, One Node: Serving GLM-5.3-FP8 on MI300X with DP8+EP8
- 14 October 2026
Serving a frontier Mixture-of-Experts model like GLM-5.3-FP8 at long context and high concurrency is a demanding workload. The model has 744 billion total parameters, with approximately 39 billion active per forward pass, and a native context limit of 1,048,576 tokens, as specified by max_position_embeddings in config.json. The most demanding serving regime combines long prompts, long generations, and many concurrent requests. That combination stresses both GPU memory and inter-GPU communication, and the layout you pick to spread the model across a node largely determines how much throughput you can sustain under that load.
DI Series: Scaling GLM-5.1-FP8 to 64 MI300X GPUs
- 21 August 2026
Serving a frontier Mixture-of-Experts (MoE) model well is a systems problem, and it gets harder the moment one node is not enough. GLM-5.1 is a good example: it is a large, sparse MoE that users want to run at long context, and it ships a new attention family that breaks assumptions older serving stacks quietly relied on. Fitting it on eight GPUs is only the start. The real question is how to keep it correct and fast as you spread it across several nodes.
ROCm 7.9 Technology Preview: ROCm Core SDK and TheRock Build System
- 20 October 2025
ROCm Core SDK 7.9.0 is a technology preview release built by TheRock. The Core SDK is a slimmed down version of the traditional ROCm release, focusing on foundational components for running high performance AI workloads on AMD GPUs. TheRock introduces a streamlined uniform system for building and testing these core components. TheRock manages dependencies between packages, so changes affecting several ROCm components can be coordinated automatically. Python releases built using TheRock can be easily installed in virtual environments, so developers can try new ROCm versions and features quickly without making system changes.