Posts by Gogineni Kailash
Using ODC to Accelerate AMD SFT Training
- 12 August 2026
Large-scale training spends a surprising share of its wall-clock time waiting instead of computing. Under Fully Sharded Data Parallel (FSDP), every layer ends in a collective all-gather or reduce-scatter, and every collective is a barrier that the whole data-parallel group has to reach together. Feed that machinery variable-length supervised fine-tuning (SFT) data and the picture gets worse: some ranks draw long documents while others draw short ones, so the fast ranks sit idle waiting for the slow ones.