Posts by Mohit Deopujari
Long Context, One Node: Serving GLM-5.3-FP8 on MI300X with DP8+EP8
- 14 October 2026
Serving a frontier Mixture-of-Experts model like GLM-5.3-FP8 at long context and high concurrency is a demanding workload. The model has 744 billion total parameters, with approximately 39 billion active per forward pass, and a native context limit of 1,048,576 tokens, as specified by max_position_embeddings in config.json. The most demanding serving regime combines long prompts, long generations, and many concurrent requests. That combination stresses both GPU memory and inter-GPU communication, and the layout you pick to spread the model across a node largely determines how much throughput you can sustain under that load.
DI Series: Scaling GLM-5.1-FP8 to 64 MI300X GPUs
- 21 August 2026
Serving a frontier Mixture-of-Experts (MoE) model well is a systems problem, and it gets harder the moment one node is not enough. GLM-5.1 is a good example: it is a large, sparse MoE that users want to run at long context, and it ships a new attention family that breaks assumptions older serving stacks quietly relied on. Fitting it on eight GPUs is only the start. The real question is how to keep it correct and fast as you spread it across several nodes.
Elevate Your LLM Inference: Autoscaling with Ray, ROCm 7.0.0, and SkyPilot
- 13 February 2026
This blog explores autoscaling of inference workloads in Ray Serve with a vLLM backend on AMD Instinct™ GPUs for large language models (LLMs). Furthermore, you will learn how to scale beyond a single cluster using SkyPilot, which enables multicloud scaling for Ray Serve. Combined with the AMD ROCm™ software platform, this creates a unified, cloud-agnostic platform that scales distributed LLM inference from single-GPU to multi-cluster deployments.
LLM Inference Optimization Using AMD GPU Partitioning
- 22 January 2026
As AI and HPC workloads grow in complexity and scale, there’s a rising need for precise GPU resource management, robust memory isolation, and efficient multi-tenant scheduling. AMD’s Instinct™ MI300 series addresses this by offering dynamic partitioning capabilities. These allow a single physical device to be segmented into multiple isolated partitions, each tailored to the needs of specific workloads. This flexibility is particularly beneficial for AI inference tasks, where different models or instances may require distinct resource allocations. Maximizing the utilization of GPU resources while ensuring that each workload operates within its own isolated environment is crucial for performance and reliability.