AMD Instinct™ Network Traffic, Congestion Trends, and Harmonics in Scale-Out Networks for AI Training Clusters

July 09, 2026 by Bryan Varble, Nithya Soundar.

6 min read. | 1414 total words.

Applications & models

AI/ML, HPC, GenAI, Performance

AI

AI training at scale, particularly for large language models (LLMs) and generative AI, demands massive computing clusters of thousands of GPUs interconnected through high-speed scale-out networks. These back-end “east-west” fabrics facilitate inter-GPU data exchange, parameter synchronization, and gradient updates — the lifeblood of distributed training frameworks like data parallelism and model parallelism.

Unlike traditional data center networks that handle diverse, asynchronous north-south traffic, AI scale-out networks carry highly synchronized, high-bandwidth flows generated by collective communication primitives — AllReduce, AllGather, and All-to-All operations from libraries like AMD RCCL (ROCm Collective Communications Library) or NVIDIA NCCL. As clusters grow to tens of thousands of GPUs, these synchronized bursts produce congestion hotspots, latency spikes, and periodic oscillations in traffic intensity that resemble harmonics in signal processing.

This post examines those traffic characteristics, congestion mechanisms, harmonic-like patterns, their impacts on training efficiency, and emerging mitigation strategies — equipping network architects and data center operators with a framework for diagnosing and addressing congestion in AI training clusters.

Architecture of AI Computing Clusters

AI clusters are structured in a hierarchical manner to support both scale-up (intra-node) and scale-out (inter-node) connectivity. Within a node or rack, GPUs communicate through high-bandwidth links like NVIDIA NVLink or AMD Infinity Fabric, offering microsecond latencies and terabytes-per-second throughput for direct memory access (DMA).

For scale-out, the back-end network spans multiple racks using protocols such as InfiniBand (IB) or RDMA over Converged Ethernet (RoCEv2). These networks employ spine-leaf topologies, often with 400 Gbps or 800 Gbps links, to minimize hops and ensure low latency. In large clusters, rail-optimized designs or multi-plane fabrics (for example, for Intel Gaudi) partition traffic to reduce contention.

Key components include:

• GPU NICs: Such as AMD Pollara, or NVIDIA ConnectX, handling RDMA operations.

• Switches: High-radix devices like Arista 7060x/77xx/78xx, Drivenets NCP/NCF, Cisco Nexus 9364, or Juniper MX/PTX/QFX.

• Topologies: Non-blocking RAIL and Tree topology fabrics to avoid oversubscription.

As clusters scale to 100K+ GPUs, the network must handle petabytes of data transfer per training epoch, making traffic management paramount. (Enfabrica, Cornelis Networks, Cisco AI/ML network challenges whitepaper)

Traffic Patterns in AI Training

AI training workloads exhibit distinct traffic characteristics compared to general-purpose computing. During model training, GPUs perform forward and backward passes to compute gradients, followed by synchronization phases where data is exchanged across the cluster.

• Bursty and Synchronized Flows: Traffic is highly bursty, with GPUs sending/receiving at full line rate (for example, 400 Gbps) simultaneously during collective syncs. These bursts repeat cyclically as GPUs complete computations and synchronize states before the next iteration. For instance, in AllReduce operations, each GPU reduces gradients locally then broadcasts results, creating synchronized waves of data. (Cisco AI/ML network challenges whitepaper)

• Elephant Flows: Most traffic consists of large; high-throughput flows with low entropy—limited IP addresses (one per NIC), UDP as the primary protocol, and fixed ports (for example, 4791 for RoCE). This results in fewer but massive flows, challenging traditional load balancing. (Cisco AI/ML network challenges whitepaper)

• East-West Dominance: Over 90% of traffic is intra-cluster (GPU-to-GPU), involving RDMA for direct memory access. In distributed training, operations like parameter sharding in model parallelism amplify inter-node exchanges. (Gallagher, LinkedIn, NADDOD)

• Low-Entropy, High-Density: Flows lack diversity in headers, leading to uneven distribution over equal-cost multipath (ECMP) routes. This is exacerbated in RoCE networks without advanced hashing. (RoCE overview)

AI/ML traffic follows a “card-game” scatter-compute-gather lifecycle: a set of computational tasks (tensors) are “dealt” to each node, processed locally, then collected through library collectives such as AllReduce or All-to-All. Because tasks are similar in size and complexity, the network follows a strict congestion-idle model — heavily loaded during scatter and gather, nearly idle during computation. Unlike the stochastic, evenly-distributed north-south traffic that traditional data centers are designed for, this synchronized pattern means congestion surges never redistribute over time, building harmonic back-pressure waves at each buffering tier: NIC, leaf switch, and spine switch. (DiVA portal)

These patterns are punishing for networks, as they diverge from the chaotic, variable traffic of internet-scale applications. (APNIC)

Congestion Trends and Causes

Congestion in AI scale-out networks arises when traffic demands exceed available capacity, leading to buffer overflows, packet drops, and latency spikes. Trends show congestion becoming chronic in clusters exceeding a few hundred GPUs, with hotspots spreading across the fabric.

• Ingress/Egress mismatches: Simultaneous bursts overwhelm switch buffers, especially when ingress traffic exceeds egress capacity. In lossless networks (for example, using priority flow control or PFC), these trigger pause frames, spreading congestion upstream. (Cisco AI/ML network challenges whitepaper, Cornelis Networks)

• Slow or stuck NICs: A GPU NIC operating below rated speed (for example, 350 Gbps on a 400 Gbps link) causes backpressure, impacting other flows through head-of-line (HoL) blocking. (Cisco AI/ML network challenges whitepaper)

• Load balancing inefficiencies: ECMP’s static hashing fails with low-entropy flows, resulting in non-uniform inter-switch link (ISL) utilization—one link at 98% while others idle. Path asymmetry in multipath routing amplifies this, degrading performance by up to 30%. (Cisco AI/ML network challenges whitepaper, Clockwork.io)

• Incast and hotspots: During AllReduce, traffic converges on shared links (incast), creating hotspots even in non-blocking fabrics. (Clockwork.io)

• RDMA recovery: Current packet loss recovery models require the “Go-back-N” retransmission model. This approach can result in much larger data re-sends and buffer consumption than a single packet-drop would incur in a traditional network. (arXiv)

• Link failures and oversubscription: Failures reduce paths, bottlenecking traffic; oversubscribed topologies cascade congestion under load. (Cornelis Networks)

Trends indicate that as cluster size grows, congestion shifts from localized to fabric-wide, with tail latencies (99th percentile) ballooning into milliseconds. (Cornelis Networks)

Harmonics and Periodic Behaviors in Network Traffic

While “harmonics” is more commonly associated with power systems or signal processing, in the context of AI network traffic, it can describe the periodic, oscillatory patterns that emerge from synchronized training workflows. These “traffic harmonics” manifest as resonant fluctuations where burst cycles align, amplifying congestion like waves interfering constructively.

• Periodic bursts: Training iterations create repetitive bursts—computation phases (low traffic) alternate with synchronization (high traffic), occurring every few seconds to minutes depending on model size. This periodicity ties to training step cycles, where AllReduce syncs generate synchronized peaks across all ports. (Cisco AI/ML network challenges whitepaper, Clockwork.io)

• Resonant effects: In large clusters, these cycles can resonate if delays align, causing cascading congestion waves. For example, power swings from synchronous GPU activity (hundreds of MW in seconds) indirectly influence network stability, though primarily in electrical domains. (arXiv)

• Frequency domain analogy: Traffic can be conceptually analyzed in the frequency domain — iteration cycles contribute a low-frequency fundamental component, while faster sub-operations like checkpointing or RDMA exchanges add higher-frequency content. Techniques such as Fourier analysis, well-established in power systems for detecting harmonics, may offer a useful analytical lens for characterizing these traffic patterns, though direct application to network traffic remains an open research area.

• Amplification in scale-out: At scale, these harmonics lead to “standing waves” of congestion, where periodic incasts overlap, exacerbating hotspots. (Clockwork.io)

Although direct literature on network harmonics is sparse, analogies from power harmonics in AI data centers (for example, distortions from switching devices) suggest similar analytical tools could apply to traffic. (arXiv, Canopywave)

Impacts on AI Training

Network inefficiencies profoundly affect training efficiency, where GPUs spend up to 32% of cycles on inter-node communication and 30% recovering from errors. (Cornelis Networks)

• Performance degradation: Congestion increases latency, stalling collective operations and idling GPUs. Tail latencies halt entire jobs, as all nodes must sync. (Cisco AI/ML network challenges whitepaper, Kentik)

• Wasted resources: Packet drops in RoCE (no TCP retransmit) trigger full RDMA retries, wasting expensive GPU cycles and extending training times by days or weeks. (Cisco AI/ML network challenges whitepaper)

• Scalability limits: In 1,000+ GPU clusters, congestion reduces model flops utilization (MFU) to 35-40%, far below theoretical peaks. (Clockwork.io)

• Economic implications: Delays postpone model deployment, inflating costs in hyperscale environments where GPU utilization averages 39-50%. (Cornelis Networks)

The harmonic congestion patterns described above compound these effects by creating predictable, recurring oscillation peaks that are difficult to mitigate with static configurations alone.

Mitigation Strategies and Solutions

Addressing these challenges requires advanced network designs and tools.

• Congestion control: Implement PFC for congestion flow detection, ECN/DCQCN for notification-based throttling, and PFC watchdogs to isolate faulty NICs. (NADDOD, Cisco AI/ML network challenges whitepaper)

• Adaptive routing and load balancing: Use fine-grained adaptive routing (for example, Cornelis Omni-Path) with telemetry for path selection, or packet spraying (NVIDIA Spectrum-X) to even flows. (Cornelis Networks, Cisco AI/ML network challenges whitepaper)

• Buffering and QoS: Shared buffers with dynamic partitioning to handle bursts; prioritize small ACKs to reduce variance. (Cisco AI/ML network challenges whitepaper)

• Credit-based flow control: Buffer-slot “credits” can be used to create and detect “threshold-crossing” events. This allows for the detection and correction of buffer “high-water” conditions programmatically, without human intervention.

• Monitoring and visibility: Tools like Cisco Nexus Dashboard or Clockwork SDF provide real-time telemetry, anomaly detection, and job-to-network correlation to preempt issues. (Cisco AI/ML network challenges whitepaper, Clockwork.io)

• Emerging technologies: SmartNICs/SuperNICs offload scheduling; Ultra Ethernet Consortium advances for better scalability. For harmonics, frequency-based analysis through ML could predict peaks. (NADDOD, ScienceDirect)

On the standards front, Ethernet congestion control has evolved through Data Center Bridging (DCB) extensions — PFC (802.1Qbb), ETS (802.1Qaz), and ECN (RFC 3168) — to provide lossless fabric capabilities previously only available in InfiniBand and Fibre Channel. The Ultra Ethernet Consortium (UEC) is advancing these further with per-packet acknowledgement and selective retransmission, targeting AI-scale deployments where Go-back-N RDMA recovery costs are prohibitive. (IEEE 802.1 DCB, IETF RFC 3168)

Summary

AI training clusters at scale face a network challenge that traditional data center designs were never built for. The synchronized, repetitive nature of collective operations like AllReduce generates bursty, wave-like congestion patterns. These surges propagate through NIC, leaf-switch, and spine-switch buffering tiers in lockstep with every training iteration — behavior that stochastic traffic models simply cannot absorb.

In this post, we examined how these harmonic congestion patterns arise, why they differ fundamentally from the random spikes traditional data centers handle, and what you can do about them:

• Understand the pattern: AI/ML traffic follows a scatter-compute-gather lifecycle that produces predictable, repeating congestion peaks — not random spikes.

• Diagnose root causes: Ingress/egress mismatches, slow NICs causing head-of-line blocking, ECMP load-balancing failures, and RDMA Go-back-N retransmission amplify the problem at scale.

• Apply layered mitigations: Deploy ECN/DCQCN for early congestion signaling, adaptive routing for per-flow or per-packet load distribution, credit-based flow control for buffer management, and real-time telemetry to correlate job behavior with network state.

Network optimization is no longer a secondary concern in AI infrastructure. It is a first-class determinant of cluster efficiency, training cost, and model iteration speed. Looking ahead, we will explore entropy-insertion techniques that stochastically de-synchronize collective phases, as well as emerging Ultra Ethernet Consortium (UEC) standards, both promising paths to breaking the harmonic cycle entirely without changes to switching hardware or routing protocols.

Additional Resources

• Ultra Ethernet Consortium

• IEEE 802.1 Data Center Bridging

• IETF RFC 3168 — The Addition of Explicit Congestion Notification (ECN) to IP

• Santos et al., “A lossless, minimal latency protocol for gigabit ATM networks,” ICNP 1998, doi:10.1109/ICNP.1998.723735

Acronyms

The following table contains acronyms and definitions used throughout this document.

Acronym	Description	Definition
AI	Artificial intelligence	Simulation of human intelligence processes by computer systems, enabling tasks that typically require human cognition such as learning, pattern recognition, and decision-making. AI systems can be narrow (task-specific) or general (broad human-like intelligence).
ATM	Asynchronous transfer mode	A high-speed networking standard that unifies voice, video, and data over fixed-size “cells,” offering predictable quality of service (QoS) through connection-oriented virtual circuits. Largely superseded by Ethernet and MPLS, though its QoS principles influence modern networking.
DCB	Data center bridging	A set of Ethernet extensions creating a lossless, unified network for data centers, combining LAN/SAN traffic onto one fabric using standards like PFC (802.1Qbb) for flow control, ETS (802.1Qaz) for bandwidth allocation, and DCBX for neighbor discovery.
FC	Fibre channel	A high-speed, reliable network technology designed primarily for storage area networks (SANs), offering lossless, low-latency data transport with security and scalability for mission-critical applications.
FCoE	Fibre channel over Ethernet	A network protocol that encapsulates fibre channel (FC) frames within Ethernet frames, allowing storage and data traffic to share the same physical network infrastructure. Requires converged network adapters (CNAs) and a lossless Ethernet fabric using priority flow control (PFC).
IB	InfiniBand	A high-performance, low-latency network interconnect standard for HPC, AI, and large data centers, enabling RDMA and high bandwidth (400 Gbps+) through a switched fabric architecture with host channel adapters (HCAs) and specialized switches.
LLM	Large language model	A neural network trained on vast amounts of text data to understand and generate human-like language. Powers tools like chatbots and content generators; excels at translation, summarization, and similar tasks.
RDMA	Remote direct memory access	A technology that allows network devices to transfer data directly to and from another system’s memory, bypassing the CPU and OS for increased speed and lower latency. Enabled through InfiniBand, RoCE, and similar protocols using host channel adapters (HCAs).
RoCE (v2)	RDMA over converged Ethernet	An advanced networking protocol running RDMA over standard IP/Ethernet networks. RoCEv2 adds IP and UDP headers, making it routable across Layer 3 networks, increasing scalability beyond a single subnet while leveraging hardware acceleration for minimal CPU involvement.

Disclaimers

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, ROCm, Instinct, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved