Reproducing AMD MLPerf Training v6.1 Submission Result#
This blog provides a step-by-step guide for reproducing AMD’s MLPerf Training v6.1 submission results. AMD submitted results on five benchmarks this round — an expanded portfolio compared to the three benchmarks submitted in v6.0:
Llama3.1-8B pretraining — full pretraining from random weights on the C4 dataset
Llama2-70B LoRA fine-tuning — parameter-efficient fine-tuning of Llama 2 70B using the GovReport dataset
DLRMv4 — recommendation model training on the Yambda dataset
Flux.1 — text-to-image diffusion model training, with two recipes: a6w4 + fp4 and fp6 + fp4
GPT-OSS-20B — pretraining of a 20-billion-parameter open-weight large language model
All AMD first-party results were submitted on AMD Instinct MI355X and MI350X GPUs. Primus, AMD’s unified training framework introduced in v6.0, powers the language-model and Flux.1 benchmarks in this submission. DLRMv4 uses the HSTU stack. Readers looking for context on the performance results and competitive analysis should refer to the companion blog post: Technical Dive into AMD’s MLPerf Training v6.1 Submission.
The table below provides a summary of AMD submissions covered in this blog:
Submission ID |
Model |
Dataset |
Convergence Target |
GPUs in This Guide |
|---|---|---|---|---|
6.1-0062 |
Llama3.1-8B Pretraining |
C4 |
Log perplexity ≤ 3.3 |
8 |
6.1-0062 |
Llama2-70B LoRA Fine-Tuning |
SCROLLS GovReport |
Cross-entropy loss ≤ 0.925 |
8 |
6.1-0061 |
DLRMv4 |
Yambda |
AUC ≥ 0.75 |
8 |
6.1-0064 |
Flux.1, a6w4 + fp4 recipe |
CC12M |
Validation loss ≤ 0.586 |
32 |
6.1-0001 |
Flux.1, fp6 + fp4 recipe |
CC12M |
Validation loss ≤ 0.586 |
32 |
6.1-0062 |
GPT-OSS-20B |
C4 |
Log perplexity ≤ 3.34 |
8 |
Prerequisites#
To follow this guide you need a Linux machine with AMD MI355X GPUs, the Docker image tag for each benchmark (see Docker Images below), and this page. For Flux.1, which runs on four nodes under Slurm, check the Slurm login node and every compute node.
Requirement |
How to check |
If it is missing |
|---|---|---|
AMD Instinct MI355X GPUs, 8 per node (4 nodes for Flux.1) |
|
If fewer than 8 appear, something is wrong with the node. Reboot it or power cycle it first |
Ubuntu 22.04 or later |
|
Install Ubuntu 22.04 or later |
amdgpu driver (versions below) |
|
Install it with the ROCm installation guide |
ROCm tools on the host |
|
Install ROCm as above; the launch scripts call this tool |
Idle GPUs |
|
Stop the other workloads |
Docker without |
|
|
Passwordless |
|
Configure it; the launch scripts use |
|
|
|
Python 3 with pip |
|
|
A writable |
|
|
Internet access |
|
Allow Docker Hub, |
Two benchmarks have extra requirements:
Llama2-70B LoRA: To download the model from HuggingFace, sign the LLAMA 2 COMMUNITY LICENSE AGREEMENT for
meta-llama/Llama-2-70b-hfand create a HuggingFace token in your token settings. Steps 2 and 4 use the token asHF_TOKEN.Flux.1: Slurm, shared storage that every node can reach, and an RDMA network between the nodes (
ls /dev/infinibandlists the RDMA devices on each node). The launcher stops if the container cannot open an RDMA device, because the job would otherwise run over TCP, too slowly to finish.
AMD’s submitted runs used the ROCm versions specified in the table below. The ROCm version is the one inside each Docker image, so the ROCm installed on your host does not need to match it.
Model |
amdgpu driver (host) |
ROCm (in the image) |
|---|---|---|
Llama3.1-8B |
6.19.4 |
7.15.0 |
Llama2-70B LoRA |
6.19.4 |
7.15.0 |
DLRMv4 |
6.19.4 |
7.2.1 |
Flux.1, a6w4 + fp4 recipe |
6.16.6 |
7.15.0 |
Flux.1, fp6 + fp4 recipe |
6.14.14 |
10.0.0 |
GPT-OSS-20B |
6.19.4 |
7.15.0 |
Note
Before every run, the launch scripts run runtime_tunables.sh on the host with sudo. It drops the filesystem caches, loads the amdgpu driver, disables CPU idle state 2, sets the CPU governor to performance, disables the NMI watchdog, NUMA balancing, and address-space randomization, and enables transparent huge pages. These settings last until the machine reboots, so use a machine that you are allowed to tune.
Disk Space, Time, and Where Things Live#
Before downloading anything, check free space with df -h on /data and on the Docker root folder (docker info --format '{{.DockerRootDir}}') to ensure you have enough space to run the benchmark.
Model |
Image |
Data and model |
Total |
|---|---|---|---|
Llama3.1-8B |
46 GB |
117 GB (85 GB dataset + 32 GB model) |
about 163 GB |
Llama2-70B LoRA |
47 GB |
about 270 GB |
about 320 GB |
DLRMv4 |
86 GB |
245 GB (227 GB dataset + 18 GB index) |
about 330 GB |
GPT-OSS-20B |
43 GB |
117 GB (85 GB dataset + 32 GB model) |
about 160 GB |
Flux.1, a6w4 + fp4 recipe |
45 GB per node |
about 2.5 TB shared, plus about 2.5 TB of local disk per node |
about 2.5 TB shared, plus about 2.55 TB on each node |
Flux.1, fp6 + fp4 recipe |
63 GB per node |
about 2.4 TB shared |
about 2.4 TB shared, plus about 63 GB on each node |
Llama3.1-8B and GPT-OSS-20B use the same dataset and tokenizer, so one download can serve both by pointing DATADIR and MODELDIR at the same location for both models.
Trial times from AMD’s submitted runs are included below for reference. A trial takes longer than its time to train because startup (model construction, data setup, and warm-up) happens before the timed region.
Model |
One trial, launch to finish |
Of which startup |
A full 10-trial set |
|---|---|---|---|
Llama3.1-8B |
about 77 min |
about 7 min |
about 13 hours |
Llama2-70B LoRA |
about 18 min |
about 11 min |
about 3 hours |
DLRMv4 |
about 25 min |
about 4 min |
about 4 hours |
GPT-OSS-20B |
about 61 min |
about 7 min |
about 10 hours |
Flux.1, a6w4 + fp4 recipe |
about 54 min |
about 6 min |
about 9.5 hours |
Flux.1, fp6 + fp4 recipe |
about 52 min |
about 4 min |
about 9 hours |
Get the Implementation Code#
The benchmark code and launch scripts are inside each Docker image, in /workspace/code. Step 1 of each benchmark copies that folder to the host (~/mlperf/<benchmark>_code, or /shared/mlperf_flux1/flux1_code for the Flux.1 a6w4 + fp4 recipe and /shared/mlperf_flux1_fp6/flux1_fp6_code for the Flux.1 fp6 + fp4 recipe), and the launch scripts run that host copy.
Docker Images#
Each benchmark has its own docker image that is available on docker hub:
Model |
Image |
Image size |
|---|---|---|
Llama2-70B LoRA |
|
47.4 GB |
Llama3.1-8B |
|
46.2 GB |
DLRMv4 |
|
85.7 GB |
Flux.1, a6w4 + fp4 recipe |
|
45.6 GB |
Flux.1, fp6 + fp4 recipe |
Built with Dockerfile ( |
63 GB |
GPT-OSS-20B |
|
42.9 GB |
If you prefer to build the docker images yourself, please follow the instruction in the Appendix.
How Results Are Checked#
All five benchmarks are scored the same way. This section explains the method once, and the final step of each benchmark shows how to apply it to that benchmark’s logs.
What the Logs Record#
Each trial writes one log, $LOGDIR/<DATESTAMP>_<N>.log, where <DATESTAMP> is the launch time as a string of digits and <N> is the trial number, which starts at 1 (Flux.1 starts at 0). The commands in each benchmark’s Step 5 use a * in place of the timestamp, so "$LOGDIR"/*_1.log selects trial 1 (*_0.log for Flux.1); change the number to read another trial. They also use $LOGDIR from Step 3, so set it again if you opened a new terminal. Scoring uses only the MLPerf log lines, which start with :::MLLOG followed by a JSON record. The following three keys in the record are important to determine the result:
run_startmarks the start of the timed region, after initialization (model construction, data setup, and warm-up) has finished.eval_accuracyholds the quality metric from one evaluation. Itsmetadata.samples_countis the number of training samples consumed up to that evaluation.run_stopmarks the end of the timed region. Itsmetadata.statusissuccesswhen an evaluation met the quality target, andabortedwhen training reached its step limit first.
For example, these are the last two records of a submitted Llama3.1-8B trial, which reached the target at its 15th evaluation:
:::MLLOG {"namespace": "", "time_ms": 1792039530706, "event_type": "POINT_IN_TIME", "key": "eval_accuracy", "value": 3.299457550048828, "metadata": {"file": "/opt/venv/lib/python3.12/site-packages/primus_mllog/mlperf_pretrain_trainer.py", "lineno": 336, "samples_count": 184320}}
:::MLLOG {"namespace": "", "time_ms": 1792039530840, "event_type": "INTERVAL_END", "key": "run_stop", "value": null, "metadata": {"file": "/opt/venv/lib/python3.12/site-packages/primus_mllog/mlperf_pretrain_trainer.py", "lineno": 216, "samples_count": 184320, "status": "success"}}
Quality, Time to Train, and Score#
To understand how MLPerf Training submissions are evaluated, it is important to understand the following metrics:
Quality. Training pauses at a fixed interval of training samples and evaluates the model on a fixed validation set. Each benchmark’s final step gives its metric, interval, and validation set. Training stops at the first evaluation that meets the target and logs
run_stopwithstatus: success.Time to train. A trial’s time to train is the
run_stoptimestamp minus therun_starttimestamp. It includes the time spent in evaluations, but not initialization.Score. A submission is a set of 10 trials. The fastest and slowest trials are dropped, and the score is the mean time to train of the remaining eight. All 10 trials should converge.
Convergence and Log Checks#
To simplify the scoring process, MLCommons provides two checkers, in the mlperf_logging package:
RCP checker. Reference convergence points (RCPs) record how many samples the reference implementation needs to converge at a given global batch size. The RCP checker compares the mean samples-to-converge of your 10-trial set against them and ends with
RCP test PASSEDorRCP test FAILED. If a set converges in fewer samples than the reference mean but is still within the allowed range, it passes, and the checker reports a normalization factor that scales the score up.Compliance checker. This checks that a trial log contains every required key with allowed values, for example the hyperparameters fixed by the rules. The Llama3.1-8B, DLRMv4, Flux.1, and GPT-OSS-20B launchers run it after every trial and append the result to
$LOGDIR/compliance_<DATESTAMP>.out, where each trial’s section ends withINFO - SUCCESSorERROR - FAILED. The Llama2-70B LoRA launcher skips it, so that benchmark’s final step shows how to run it on the host.
Install the checkers on the host once:
python3 -m pip install "git+https://github.com/mlcommons/logging.git@6.1.0-rc2"
The RCP checker reads only files named result_<N>.txt, so each benchmark’s final step copies the trial logs to those names before running it. It needs a full set of 10 trials: with fewer logs it prints nan and RCP test FAILED, so skip it after a single test trial. Use a fresh $LOGDIR for each 10-trial set, so that the glob "$LOGDIR"/*_[0-9]*.log matches only that set.
Benchmark 1: Llama3.1-8B Pretraining#
Background - Llama3.1-8B#
The Llama3.1-8B pretraining benchmark measures how quickly a model can be trained from random weights to reach a target validation loss on the C4 (Colossal Cleaned Common Crawl) dataset. AMD led the development of this benchmark in MLPerf Training. No checkpoint conversion is required. Disk space required: ~85 GB dataset + ~32 GB model directory.
Step 1: Pull the Docker Image and Copy the Code - Llama3.1-8B#
mkdir -p ~/mlperf && cd ~/mlperf
docker pull rocm/amd-mlperf:llama31_8b_training_6.1
container_id=$(docker create rocm/amd-mlperf:llama31_8b_training_6.1) && \
docker cp $container_id:/workspace/code ./llama31_8b_code && \
docker rm $container_id
To build the image instead, see the Build the Images Instead of Pulling section at the end of this guide.
Step 2: Prepare Dataset and Tokenizer - Llama3.1-8B#
mkdir -p /data/mlperf_llama31_8b
cd /data/mlperf_llama31_8b
# Download training data
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d data https://training.mlcommons-storage.org/metadata/llama-3-1-8b-preprocessed-c4-dataset.uri
# Download model tokenizer
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d model https://training.mlcommons-storage.org/metadata/llama-3-1-8b-tokenizer.uri
After the download is complete, the data directory contains files ending in both .idx and .bin with the following naming conventions:
Training partitions:
c4-train.en_6_text_documentValidation partitions:
c4-validation-91205-samples.en_text_document
Step 3: Set Up Environment - Llama3.1-8B#
Enter the llama31_8b_code folder that you copied in Step 1, then set the paths and the image:
cd ~/mlperf/llama31_8b_code
mkdir -p /data/mlperf_llama31_8b/results
export DATADIR=/data/mlperf_llama31_8b/data
export MODELDIR=/data/mlperf_llama31_8b/model/
export LOGDIR=/data/mlperf_llama31_8b/results
export CONT=rocm/amd-mlperf:llama31_8b_training_6.1
sudo chmod -R 777 $LOGDIR
Source the MI355X configuration file:
source config_MI355X_1x8x1.sh
The source command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.
Step 4: Run Training - Llama3.1-8B#
Run one trial first to confirm that everything works. It takes about 77 minutes, including about 7 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows ENDING TIMING RUN followed by RESULT,LLAMA3.1_8B,,<seconds>,AMD,<start time>, the total run time in seconds including startup.
export NEXP=1
bash run_with_docker.sh
For a compliant 10-run submission (about 13 hours):
export NEXP=10
bash run_with_docker.sh
After completion, the trial logs are in $LOGDIR.
Step 5: Check Quality and Validate Results - Llama3.1-8B#
Quality is validation log perplexity on held-out C4 text (lower is better), evaluated every 12,288 training sequences. A trial succeeds at the first evaluation at or below 3.3.
Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/*_1.log
The trial converged if the last eval_accuracy value is 3.3 or lower and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. Across the 10 submitted trials, the target was met at the 15th or 16th evaluation (184,320–196,608 samples), with times to train of 69.9–74.8 minutes.
The last throughput record gives the average training throughput; expect approximately 44 samples/second on MI355X:
grep overall_throughput "$LOGDIR"/*_1.log | tail -1
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored 72.94 minutes. Then run the RCP checker, after copying each trial log to the result_<N>.txt name it reads:
i=0; for f in "$LOGDIR"/*_[0-9]*.log; do cp "$f" "$LOGDIR/result_$((i++)).txt"; done
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
Finally, confirm that every trial’s section of $LOGDIR/compliance_*.out ends with INFO - SUCCESS.
Benchmark 2: Llama2-70B LoRA Fine-Tuning#
Background - Llama2-70B LoRA#
The Llama2-70B LoRA fine-tuning benchmark measures how quickly the Llama 2 70B model can be fine-tuned to reach the target cross-entropy loss on the SCROLLS GovReport summarization dataset (pre-tokenized, packed to sequence length 8,192), using Low-Rank Adaptation (LoRA). Disk space required: ~270 GB for model download and conversion.
Downloading the model needs a HuggingFace token (HF_TOKEN) with access to meta-llama/Llama-2-70b-hf; see Prerequisites.
Step 1: Pull the Docker Image and Copy the Code - Llama2-70B LoRA#
mkdir -p ~/mlperf && cd ~/mlperf
docker pull rocm/amd-mlperf:llama2_70b_training_6.1
container_id=$(docker create rocm/amd-mlperf:llama2_70b_training_6.1) && \
docker cp $container_id:/workspace/code ./llama2_70b_code && \
docker rm $container_id
To build the image instead, see the Build the Images Instead of Pulling section at the end of this guide.
Step 2: Download and Preprocess Data - Llama2-70B LoRA#
Start a container with the host download directory mounted at /data. The preparation script is the copy in the image. The commands below are run in the shell you start in, so the folder you are in does not matter:
mkdir -p /data/mlperf_llama2/data
docker run --rm -it -v /data/mlperf_llama2/data:/data \
--net=host --uts=host \
--ipc=host --device /dev/dri --device /dev/kfd \
--security-opt=seccomp=unconfined \
rocm/amd-mlperf:llama2_70b_training_6.1
From inside the container, run the preparation script prepare_data_and_model.sh:
export HF_TOKEN=<your_huggingface_token>
bash ./scripts/prepare_data_and_model.sh
Exit the container. The data directory on the host should look like this:
/data/mlperf_llama2/data
├── train.npy # Packed training data (seq_length=8192)
├── validation.npy # Packed validation data
├── packed_metadata.jsonl # Sequence metadata for Megatron-Bridge
└── megatron_checkpoints/
└── Llama-2-70b-hf/
├── latest_checkpointed_iteration.txt
├── latest_train_state.pt
└── iter_0000000/
├── __0_0.distcp # Sharded weights (~64 GB)
├── __0_1.distcp # Sharded weights (~64 GB)
├── common.pt
├── .metadata
├── metadata.json
├── run_config.yaml
├── modelopt_run_config.yaml
├── train_state.pt
└── tokenizer/
├── tokenizer.model
├── tokenizer_config.json
└── special_tokens_map.json
Step 3: Set Up Environment - Llama2-70B LoRA#
Enter the llama2_70b_code folder that you copied in Step 1, then set the paths and the image:
cd ~/mlperf/llama2_70b_code
mkdir -p /data/mlperf_llama2/results
export DATADIR=/data/mlperf_llama2
export MODELDIR=/data/mlperf_llama2/data
export LOGDIR=/data/mlperf_llama2/results
export CONT=rocm/amd-mlperf:llama2_70b_training_6.1
sudo chmod -R 777 $LOGDIR
Source the MI355X configuration:
source config_MI355X_1x8x1.sh
The source command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.
Step 4: Run Training - Llama2-70B LoRA#
Run one trial first to confirm that everything works. It takes about 18 minutes, including about 11 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows ENDING TIMING RUN followed by RESULT,LLAMA2_70B_LORA_SFT,,<seconds>,AMD,<start time>, the total run time in seconds including startup.
export HF_TOKEN=<your_huggingface_token>
export NEXP=1
bash run_with_docker.sh
For a compliant 10-run submission (about 3 hours):
export HF_TOKEN=<your_huggingface_token>
export NEXP=10
bash run_with_docker.sh
After completion, the trial logs are in $LOGDIR.
Step 5: Check Quality and Validate Results - Llama2-70B LoRA#
Quality is the cross-entropy loss on the GovReport validation set (lower is better), evaluated every 384 training samples after the first three evaluations are skipped. A trial succeeds at the first evaluation at or below 0.925.
Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/*_1.log
The trial converged if the last eval_accuracy value is 0.925 or lower and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. All 10 submitted trials met the target at the fifth evaluation (3,072 samples), with a final loss of 0.907–0.913 and times to train of 6.84–6.86 minutes.
The last throughput record gives the average training throughput; expect approximately 7.5 samples/second on MI355X:
grep '"key": "throughput"' "$LOGDIR"/*_1.log | tail -1
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored 6.98 minutes. Then run the RCP checker, after copying each trial log to the result_<N>.txt name it reads:
i=0; for f in "$LOGDIR"/*_[0-9]*.log; do cp "$f" "$LOGDIR/result_$((i++)).txt"; done
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
The launcher does not run the compliance checker, so run it on the host for each trial log. Every trial should print INFO - SUCCESS:
for f in "$LOGDIR"/*_[0-9]*.log; do
python3 -m mlperf_logging.compliance_checker --usage training --ruleset 6.1.0 "$f" 2>&1 | grep -E "SUCCESS|FAILED"
done
Benchmark 3: DLRMv4#
Background - DLRMv4#
DLRMv4 is the latest generation of the Deep Learning Recommendation Model benchmark in MLPerf Training. It trains an HSTU ranking model on the Yambda-5b music-recommendation dataset and is one of the computationally heaviest benchmarks in the suite due to its large embedding tables and high memory requirements. Disk space required: ~227 GB for the prepared dataset, plus an ~18 GB time-window index built before the first run.
Note: DLRMv4 in MLPerf Training v6.1 uses the Yambda dataset — a change from the Criteo Terabyte dataset used by prior DLRM benchmarks. The prepared dataset is published by MLCommons at
dlrmv4_dataset.uri.
Step 1: Pull the Docker Image and Copy the Code - DLRMv4#
mkdir -p ~/mlperf && cd ~/mlperf
docker pull rocm/amd-mlperf:dlrmv4_training_6.1
container_id=$(docker create rocm/amd-mlperf:dlrmv4_training_6.1) && \
docker cp $container_id:/workspace/code ./dlrmv4_code && \
docker rm $container_id
To build the image instead, see the Build the Images Instead of Pulling section at the end of this guide.
Step 2: Prepare Dataset - DLRMv4#
mkdir -p /data/mlperf_dlrmv4
cd /data/mlperf_dlrmv4
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d data https://training.mlcommons-storage.org/metadata/dlrmv4_dataset.uri
After the download is complete, the data directory should contain the following folders:
Training cache:
processed_5b/hstu_cache_L4086/Item popularity:
processed_5b/item_popularity.npyItem mappings:
shared_metadata/album_item_mapping.parquetandshared_metadata/artist_item_mapping.parquet
Verify the checksums and build the time-window index once before the first run. The first command runs from the dlrmv4_code folder that you copied in Step 1:
cd ~/mlperf/dlrmv4_code
DLRM_DATA_PATH=/data/mlperf_dlrmv4/data bash scripts/verify_dataset.sh
docker run --rm -v /data/mlperf_dlrmv4/data:/data \
rocm/amd-mlperf:dlrmv4_training_6.1 \
python3 -m generative_recommenders.dlrm_v4.datasets.offline_ts_indices \
--cache-dir /data/processed_5b/hstu_cache_L4086 \
--history-length 4086 --min-history 4086 --window-seconds 86400
Step 3: Set Up Environment - DLRMv4#
Enter the dlrmv4_code folder that you copied in Step 1, then set the paths and the image:
cd ~/mlperf/dlrmv4_code
mkdir -p /data/mlperf_dlrmv4/results
export DATADIR=/data/mlperf_dlrmv4/data
export LOGDIR=/data/mlperf_dlrmv4/results
export CONT=rocm/amd-mlperf:dlrmv4_training_6.1
sudo chmod -R 777 $LOGDIR
Source the MI355X configuration:
source config_MI355X_1x8x1.sh
The source command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.
Step 4: Run Training - DLRMv4#
Run one trial first to confirm that everything works. It takes about 25 minutes, including about 4 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows ENDING TIMING RUN followed by RESULT,DLRMV4,,<seconds>,AMD,<start time>, the total run time in seconds including startup.
export NEXP=1
bash run_with_docker.sh
For a compliant 10-run submission (about 4 hours):
export NEXP=10
bash run_with_docker.sh
To run the same single-node configuration through Slurm instead, edit job.sh to set --partition, CONT, DATADIR, and LOGDIR, then submit it. job.sh is set for 1 node and DGXSYSTEM=MI355X_1x8x1, and runs 10 trials by default (NEXP=10). CONT must be an image every node can pull. Run the following Slurm command to submit the job:
sbatch job.sh
Step 5: Check Quality and Validate Results - DLRMv4#
Quality is the AUC of the model’s listen_plus prediction on a held-out time window (higher is better), evaluated every 2,293,760 training samples after the first 57,344,000. A trial succeeds at the first evaluation at or above 0.75.
Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/*_1.log
The trial converged if the last eval_accuracy value is 0.75 or higher and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. Convergence varies more between trials than in the other benchmarks: across the 10 submitted trials, the target was met after 4 to 15 evaluations (64,225,280–89,456,640 samples), with times to train of 16.5–25.5 minutes.
The last throughput record gives the average training throughput; expect approximately 62,400 samples/second on MI355X:
grep overall_throughput "$LOGDIR"/*_1.log | tail -1
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored 18.95 minutes. Then run the RCP checker, after copying each trial log to the result_<N>.txt name it reads:
i=0; for f in "$LOGDIR"/*_[0-9]*.log; do cp "$f" "$LOGDIR/result_$((i++)).txt"; done
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
Finally, confirm that every trial’s section of $LOGDIR/compliance_*.out ends with INFO - SUCCESS.
Benchmark 4: Flux.1#
Background - Flux.1#
Flux.1 is the text-to-image diffusion model benchmark, new to AMD’s MLPerf Training submissions in the v6.1 round. It measures how quickly the 12-billion-parameter Flux.1 Schnell model can be trained to reach the target validation loss (≤ 0.586) on the CC12M dataset, using 32 GPUs (4 nodes × 8 MI355X GPUs). The model has 19 double-stream (MMDiT) layers and 38 single-stream layers, with a hidden size of 3,072 and 24 attention heads.
This is one benchmark with two training recipes, which differ in the precision of the forward GEMMs of the transformer’s linear layers. Both recipes run all backward GEMMs in MXFP4:
a6w4 + fp4 recipe: the forward GEMMs use the a6w4 format. It runs from the published image
rocm/amd-mlperf:mlperf_flux1_6.1.fp6 + fp4 recipe: runs the forward GEMMs in MXFP6 on Primus’s Megatron-LM backend, while attention, normalization and communication run in BF16. It has no published image, so Step 1 builds it.
Pick one recipe and follow its steps; they are independent, and each has its own code folder, data, and results. The a6w4 + fp4 recipe needs about 2.5 TB on shared storage, plus about 2.5 TB of local disk per node. The fp6 + fp4 recipe needs about 2.4 TB on shared storage.
Recipe A: a6w4 + fp4#
Step 1: Pull the Docker Image and Copy the Code - Flux.1 (a6w4 + fp4)#
mkdir -p /shared/mlperf_flux1 && cd /shared/mlperf_flux1
docker pull rocm/amd-mlperf:mlperf_flux1_6.1
container_id=$(docker create rocm/amd-mlperf:mlperf_flux1_6.1) && \
docker cp $container_id:/workspace/code ./flux1_code && \
docker rm $container_id
The job runs its scripts from flux1_code on every node, so /shared/mlperf_flux1 must be shared storage that every node can reach; replace it with your own shared path in every Flux.1 command. Every node also needs the image. Pull it on each node, for example with srun --nodes=4 --ntasks-per-node=1 docker pull rocm/amd-mlperf:mlperf_flux1_6.1 (add your usual --partition option). To build the image instead, see the Build the Images Instead of Pulling section at the end of this guide.
Step 2: Prepare Dataset - Flux.1 (a6w4 + fp4)#
Download the preprocessed data to shared storage, running the script from the flux1_code folder that you copied in Step 1:
cd /shared/mlperf_flux1/flux1_code
PREPROCESSED_PATH=/shared/mlperf_flux1/data ./scripts/download_ref_data.sh
The data directory must contain the three downloaded directories, which training reads as-is:
/shared/mlperf_flux1/data/
├── cc12m_preprocessed
├── coco_preprocessed
└── empty_encodings
The script job.sh copies the data to /data/mlperf_flux1 on each node (override with FLUX_LOCAL_DATADIR), so DATADIR must be set to a different path from /data/mlperf_flux1.
Step 3: Set Up Environment - Flux.1 (a6w4 + fp4)#
Enter the flux1_code folder that you copied in Step 1, then set the paths and the image:
cd /shared/mlperf_flux1/flux1_code
mkdir -p /shared/mlperf_flux1/results
export DATADIR=/shared/mlperf_flux1/data
export LOGDIR=/shared/mlperf_flux1/results
export CONT=rocm/amd-mlperf:mlperf_flux1_6.1
sudo chmod -R 777 $LOGDIR
LOGDIR must also be on shared storage. sudo chmod -R 777 $LOGDIR lets the container write its logs there.
Source the 4-node (32-GPU) MI355X configuration:
source config_MI355X_4x8x32.sh
The source command loads the settings, including NNODES, into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.
Step 4: Run Training - Flux.1 (a6w4 + fp4)#
Edit the #SBATCH header in job.sh to set your partition (sinfo lists the partitions you can use; replace <your_partition> with one of them), and uncomment --nodelist if you need specific nodes:
cd /shared/mlperf_flux1/flux1_code
sed -i 's/--partition=YOUR_PARTITION/--partition=<your_partition>/' job.sh
Submit from the flux1_code folder, in the same shell where you sourced the configuration, so that the job inherits NNODES and the other settings. Run one trial first to confirm that everything works:
export NEXP=1
sbatch --nodes=$NNODES job.sh
sbatch prints Submitted batch job <jobid> and returns at once; the job keeps running if you log out. Check its state with squeue -u $USER.
For a compliant 10-trial submission:
export NEXP=10
sbatch --nodes=$NNODES job.sh
The job runs in three stages:
Stage the dataset. Each node copies
cc12m_preprocessed,coco_preprocessed, andempty_encodingsfromDATADIRto its local disk at/data/mlperf_flux1(override withFLUX_LOCAL_DATADIR), then writes a marker that records the source. Files already present with the same size are skipped, so only the first job copies the full ~2.5 TB. Training refuses to start on a node without a complete copy, soFLUX_STAGE_LOCAL=0, which skips this stage, only works when every node is already staged.Train. Each node starts one container, runs
runtime_tunables.shto tune the host, checks that an RDMA device is usable, and runs theNEXPtrials back to back in that container.Record. After each trial, the first node writes that trial’s MLPerf records to
$LOGDIR/<DATESTAMP>_<N>.log, numbered from 0, and runs the compliance checker.
Slurm writes the job’s console output to flux1_primus_<jobid>.out and flux1_primus_<jobid>.err in the submission directory. The script job.sh requests a 60-hour time limit, which covers staging plus 10 trials of about 54 minutes each.
Step 5: Check Quality and Validate Results - Flux.1 (a6w4 + fp4)#
Quality is the diffusion validation loss on 29,696 COCO-2014 samples (lower is better), evaluated every 262,144 training samples. A trial succeeds at the first evaluation at or below 0.586.
Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/*_0.log
The trial converged if the last eval_accuracy value is 0.586 or lower and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. In AMD’s 4-node runs, converged trials met the target after 33 to 35 evaluations (8,650,752–9,175,040 samples), taking 47 to 50 minutes.
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight. Then run the RCP checker, after copying each trial log to the result_<N>.txt name it reads:
i=0; for f in "$LOGDIR"/*_[0-9]*.log; do cp "$f" "$LOGDIR/result_$((i++)).txt"; done
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
Finally, confirm that every trial’s section of $LOGDIR/compliance_*.out ends with INFO - SUCCESS.
Recipe B: fp6 + fp4#
The network settings are written for the AMD Pensando™ Pollara 400 AI NICs (RoCE v2, one NIC per GPU) that AMD’s runs used; Step 3 shows where to change them for other RDMA NICs.
Step 1: Get the Code and Build the Docker Image - Flux.1 (fp6 + fp4)#
This recipe has no published image, so build it from the submission code in the MLCommons results repository. Copy the implementation folder to shared storage:
mkdir -p /shared/mlperf_flux1_fp6 && cd /shared/mlperf_flux1_fp6
git clone https://github.com/mlcommons/training_results_v6.1.git
cp -r training_results_v6.1/AMD/benchmarks/flux1/implementations/4xMI355X_EPYC_9575F_primus_megatron ./flux1_fp6_code
The job runs its scripts from flux1_fp6_code on every node, so /shared/mlperf_flux1_fp6 must be on shared storage that every node can reach; replace it with your own shared path in every command below.
Every node needs the image. Build it on all four nodes at once (add your usual --partition option):
srun --nodes=4 --ntasks-per-node=1 \
bash -c 'cd /shared/mlperf_flux1_fp6/flux1_fp6_code && docker build -t mlperf_flux1_fp6:6.1 .'
The Dockerfile starts from the public rocm/primus:v26.7 image and installs AITER, Primus-Turbo and Primus at pinned commits. The build does not need a GPU. It also compiles the AITER kernels that training uses into the image, so nothing is compiled during a trial. The image records the commits it was built from in /opt/mlperf/stack.txt.
Step 2: Prepare Dataset - Flux.1 (fp6 + fp4)#
Run the following commands to download the dataset, convert it to the format training reads, and verify it:
mkdir -p /shared/mlperf_flux1_fp6/data
docker run --rm -v /shared/mlperf_flux1_fp6/data:/data mlperf_flux1_fp6:6.1 \
bash /workspace/code/scripts/prepare_data.sh
docker run --rm -v /shared/mlperf_flux1_fp6/data:/data mlperf_flux1_fp6:6.1 \
bash /workspace/code/scripts/verify_data.sh /data/mlperf_flux1
The script prepare_data.sh downloads the MLCommons-preprocessed CC12M training set, the COCO-2014 evaluation subset and the empty encodings, and converts them to Megatron-Energon WebDataset shards in one streaming pass. It downloads four files at a time, converts each into one shard and deletes it, and keeps at most six downloaded files (about 1.2 GB) on disk at once. So the peak disk use is the final dataset size plus about 1.2 GB, not a raw download plus a converted copy. If the script stops partway, run it again: it skips the shards that already exist. After it completes, the data directory should contain the following:
Training:
mlperf_flux1/train/(4,762 shards, 1,099,776 samples)Validation:
mlperf_flux1/val/(129 shards, 29,696 samples)Empty encodings:
mlperf_flux1/empty_encodings/Energon metadata:
mlperf_flux1/.nv-meta/
The script verify_data.sh checks these counts, that every validation sample has its evaluation timestep, and the checksum of every shard against the manifest shipped with the code. It ends with [verify_data] PASS. If a shard fails its checksum, for example one cut short when an earlier run of prepare_data.sh stopped, delete that shard and run prepare_data.sh again.
Every node reads the data from shared storage at a rate of about 1.5 GB/s per node. If your shared storage is slower, copy mlperf_flux1 to the same local path on every node and set DATADIR there in Step 3.
Step 3: Set Up Environment - Flux.1 (fp6 + fp4)#
The script job.sh sets the image and the paths itself, so set them in that file rather than in your shell. Enter the flux1_fp6_code folder that you copied in Step 1, then set the paths and the image:
cd /shared/mlperf_flux1_fp6/flux1_fp6_code
mkdir -p /shared/mlperf_flux1_fp6/results
export LOGDIR=/shared/mlperf_flux1_fp6/results
sed -i \
-e 's|^export CONT=.*|export CONT=mlperf_flux1_fp6:6.1|' \
-e 's|^export DATADIR=.*|export DATADIR=/shared/mlperf_flux1_fp6/data|' \
-e "s|^export LOGDIR=.*|export LOGDIR=$LOGDIR|" \
-e 's|^export CACHEDIR=.*|export CACHEDIR=/tmp/mlperf_flux1_fp6_cache|' \
-e 's|--partition=YOUR_PARTITION|--partition=<your_partition>|' \
-e '/^export CONT=/a export CONT_PULL=0' \
job.sh
sudo chmod -R 777 $LOGDIR
The environment variable LOGDIR must be on shared storage, and CACHEDIR holds the compile caches on each node’s local disk. CONT_PULL=0 makes the launcher use the image you built instead of pulling it. sinfo lists the partitions you can use. job.sh uses the 4-node (32-GPU) configuration config_MI355X_04x08x32_mxfp6heal_hb128.sh, so there is no configuration to source.
The network settings are in config_common.sh. If your nodes do not have eight Pollara AI NICs named ionic_0 through ionic_7, set NCCL_IB_HCA to your RDMA devices (ls /sys/class/infiniband lists them) and NCCL_IB_GID_INDEX to their RoCE v2 GID index. If your site provides an RCCL topology file for the nodes, set RCCL_TOPO_FILE in job.sh to its path on the host.
Step 4: Run Training - Flux.1 (fp6 + fp4)#
Run one trial first to confirm that everything works. It takes about 52 minutes, including about 4 minutes of startup before the timed run begins:
cd /shared/mlperf_flux1_fp6/flux1_fp6_code
NEXP=1 sbatch job.sh
For a compliant 10-trial submission (about 9 hours), submit the job with:
sbatch job.sh
sbatch prints Submitted batch job <jobid> and returns at once; check the job’s state with squeue -u $USER. Slurm writes the job’s console output to flux1_<jobid>.out and flux1_<jobid>.err in the submission folder, and each node writes its console output for trial <N> to $LOGDIR/MI355X_04x08x32_mxfp6heal_hb128_run<N>_node<rank>.log. A trial is finished when that output shows ENDING TIMING RUN followed by a line of the form RESULT,FLUX1,<seed>,<seconds>,AMD,<start time>, the total run time including startup.
Note
Before the first trial, the launcher runs runtime_tunables.sh, which logs the host settings and warns where they differ from the ones AMD’s runs used; it does not change them. Before every trial, it drops the page cache, with passwordless sudo or through the privileged container, and does not start the trial if it cannot.
Step 5: Check Quality and Validate Results - Flux.1 (fp6 + fp4)#
Quality is the diffusion validation loss on 29,696 COCO-2014 samples (lower is better), evaluated every 262,144 training samples. A trial succeeds at the first evaluation at or below 0.586.
This submission writes each trial’s MLPerf log to $LOGDIR/result_<N>.txt, numbered from 0. Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/result_0.txt
The trial converged if the last eval_accuracy value is 0.586 or lower and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. In AMD’s 10 submitted trials, the target was met after 32 to 36 evaluations (8,388,608–9,437,184 samples), taking 45.0 to 50.3 minutes.
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored 47.11 minutes. The logs already have the names the RCP checker reads, so run it on $LOGDIR directly:
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
Finally, confirm that every trial’s compliance file ends with INFO - SUCCESS:
tail -n 1 "$LOGDIR"/raw/*_node0/compliance_*.out
Benchmark 5: GPT-OSS-20B#
Background - GPT-OSS-20B#
GPT-OSS-20B is a 20-billion-parameter Mixture-of-Experts language model pretraining benchmark new to AMD’s MLPerf Training portfolio in v6.1, trained on the C4 dataset to a target log perplexity of ≤ 3.34. Disk space required: ~85 GB dataset + ~32 GB model directory.
Step 1: Pull the Docker Image and Copy the Code - GPT-OSS-20B#
mkdir -p ~/mlperf && cd ~/mlperf
docker pull rocm/amd-mlperf:gpt_oss_20b_training_6.1
container_id=$(docker create rocm/amd-mlperf:gpt_oss_20b_training_6.1) && \
docker cp $container_id:/workspace/code ./gpt_oss_20b_code && \
docker rm $container_id
To build the image instead, see the Build the Images Instead of Pulling section at the end of this guide.
Step 2: Prepare Dataset - GPT-OSS-20B#
Run the following commands to download the dataset and tokenizer:
mkdir -p /data/gpt_oss_20b
cd /data/gpt_oss_20b
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d data https://training.mlcommons-storage.org/metadata/llama-3-1-8b-preprocessed-c4-dataset.uri
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d model https://training.mlcommons-storage.org/metadata/llama-3-1-8b-tokenizer.uri
After the download is complete, the data directory should contain the following:
Training:
c4-train.en_6_text_document.binand.idxValidation:
c4-validation-91205-samples.en_text_document.binand.idx
Step 3: Set Up Environment - GPT-OSS-20B#
Enter the gpt_oss_20b_code folder that you copied in Step 1, then set the paths and the image:
cd ~/mlperf/gpt_oss_20b_code
mkdir -p /data/gpt_oss_20b/results
export DATADIR=/data/gpt_oss_20b/data
export MODELDIR=/data/gpt_oss_20b/model
export LOGDIR=/data/gpt_oss_20b/results
export CONT=rocm/amd-mlperf:gpt_oss_20b_training_6.1
sudo chmod -R 777 $LOGDIR
Source the MI355X configuration:
source config_MI355X_1x8x1_tp1pp1ep1_gbs32.sh
The source command loads the settings into your current shell. Run Step 4 in this same shell, or repeat this step in a new terminal.
Step 4: Run Training - GPT-OSS-20B#
Run one trial first to confirm that everything works. It takes about 61 minutes, including about 7 minutes of startup. The console can stay quiet while training. The trial is finished when the output shows ENDING TIMING RUN followed by RESULT,GPT_OSS_20B,,<seconds>,AMD,<start time>, the total run time in seconds including startup.
export NEXP=1
bash run_with_docker.sh
For a compliant 10-run submission (about 10 hours):
export NEXP=10
bash run_with_docker.sh
Logs default to MLPerf records and run summaries. Set MLPERF_VERBOSE_LOGS=1 for debugging.
Step 5: Check Quality and Validate Results - GPT-OSS-20B#
Quality is validation log perplexity on held-out C4 text (lower is better), evaluated every 12,288 training sequences. A trial succeeds at the first evaluation at or below 3.34.
Print the trial’s start, evaluation, and stop records:
grep -E '"key": "(run_start|eval_accuracy|run_stop)"' "$LOGDIR"/*_1.log
The trial converged if the last eval_accuracy value is 3.34 or lower and run_stop shows "status": "success". Its time to train is the run_stop time_ms minus the run_start time_ms, in milliseconds. Across the 10 submitted trials, the target was met at the 18th or 19th evaluation (221,184–233,472 samples), with times to train of 54.0–57.0 minutes.
The last throughput record gives the average training throughput; expect approximately 68 samples/second on MI355X:
grep overall_throughput "$LOGDIR"/*_1.log | tail -1
To score a 10-trial set, drop the fastest and slowest trials and average the time to train of the other eight; the AMD submission scored 56.18 minutes. Then run the RCP checker, after copying each trial log to the result_<N>.txt name it reads:
i=0; for f in "$LOGDIR"/*_[0-9]*.log; do cp "$f" "$LOGDIR/result_$((i++)).txt"; done
python3 -m mlperf_logging.rcp_checker "$LOGDIR" --rcp_version 6.1.0
Finally, confirm that every trial’s section of $LOGDIR/compliance_*.out ends with INFO - SUCCESS.
Troubleshooting#
Here is a list of common issues and their resolutions:
permission deniedwhen runningdocker: Your user is not in thedockergroup. Runsudo usermod -aG docker $USER, then log out and back in.sudo: a password is requiredin the console output: The launch scripts cannot type a password. Configure passwordlesssudofor your user, then start the run again.cpupower: command not found: Install it withsudo apt-get install linux-tools-common linux-tools-$(uname -r).docker pullfails withunauthorizedorpull access denied: Check that the image tag is spelled exactly as in the Docker Images table, and rundocker loginif your account needs it.A download stops partway: Run the same download command again from the same folder. It resumes partial files and verifies checksums when it finishes.
No space left on device: Compare the sizes in Disk Space, Time, and Where Things Live withdf -hon the Docker root folder and on/data. Free space, then start again.Llama2-70B LoRA download fails with 401 or 403: The HuggingFace token is wrong, or the account has not accepted the license for
meta-llama/Llama-2-70b-hf.The run ends within a minute or two: Another job may be using the GPUs. Check
amd-smi monitoranddocker ps, and read the console output for the first error.The console is quiet for minutes: This is normal. Startup takes several minutes, and training prints only when it evaluates the model. Check that the container is still running with
docker ps.Starting over, or changing a script or configuration: The launcher runs the copy in the code folder (
~/mlperf/<folder>,/shared/mlperf_flux1/flux1_codefor the Flux.1 a6w4 + fp4 recipe, or/shared/mlperf_flux1_fp6/flux1_fp6_codefor the Flux.1 fp6 + fp4 recipe), not the one inside the image, so edit that copy. To start from a clean copy, delete the folder withsudo rm -rf(the container writes files as root), then repeat Step 1. Repeating Step 1 into a folder that already exists nests a second copy inside it and keeps the old files.Stopping a run: Press Ctrl+C, then find the container with
docker psand remove it withdocker rm -f <container name>. For a Flux.1 job, runscancel <jobid>.Flux.1 stops at startup with an RDMA message: The container could not open an RDMA device. Check
ls /dev/infinibandon every node and that the nodes are connected to the RDMA network.
Expected Results#
The time-to-train values for the MLPerf Training v6.1 submissions by AMD are included below for reference. You should achieve similar results by following the procedures described above. Keep in mind that slight variations in hardware setups and conditions may lead to minor discrepancies from AMD’s results.
Submission ID |
Model |
MI355X TTT (min) |
|---|---|---|
6.1-0062 |
Llama3.1-8B Pretraining |
72.94 |
6.1-0062 |
Llama2-70B LoRA Fine-Tuning |
6.98 |
6.1-0061 |
DLRMv4 |
18.95 |
6.1-0062 |
GPT-OSS-20B |
56.18 |
6.1-0064 |
Flux.1, a6w4 + fp4 recipe (4 nodes, 32 GPUs) |
47.18 |
6.1-0001 |
Flux.1, fp6 + fp4 recipe (4 nodes, 32 GPUs) |
47.11 |
Summary#
This guide covers the full reproduction workflow for AMD’s MLPerf Training v6.1 submission across five benchmarks — Llama3.1-8B pretraining, Llama2-70B LoRA fine-tuning, DLRMv4, Flux.1 (two recipes, a6w4 + fp4 and fp6 + fp4), and GPT-OSS-20B — on AMD Instinct MI355X GPUs. All benchmarks follow the same three-phase pattern:
Pull the Docker image and copy the code out of it (the Flux.1 fp6 + fp4 recipe builds its image from the MLCommons code instead).
Prepare the dataset and source the GPU-specific configuration file.
Run and check each trial’s quality and time to train from its MLPerf log, then score the 10-trial set and run the MLCommons RCP and compliance checkers.
This round marks a significant expansion of AMD’s MLPerf Training portfolio — from three benchmarks in v6.0 to five in v6.1 — covering language model pretraining and fine-tuning, recommendation systems, and generative image modeling. Dive into the MLPerf Training v6.1 blog to understand the optimizations behind these submissions, then put them into practice on AMD Instinct™ GPUs with ROCm.
Appendix: Build the Docker Images Instead of Pulling#
AMD’s submission code is also published in the MLCommons results repository. Each implementation folder there contains its Dockerfile and every file that the Dockerfile copies, so you can build an image from it instead of pulling one:
git clone https://github.com/mlcommons/training_results_v6.1.git
cd training_results_v6.1/AMD
Run the build from that benchmark’s implementation folder, and tag it with the image name from the Docker Images table so that CONT and the other commands in this guide work unchanged. If you build an image, skip the docker pull in that benchmark’s Step 1 and copy the code from your image with the same docker create and docker cp commands, or run the benchmark from the implementation folder itself.
Model |
Implementation folder |
Build command |
|---|---|---|
Llama3.1-8B |
|
|
Llama2-70B LoRA |
|
|
DLRMv4 |
|
|
Flux.1, a6w4 + fp4 recipe |
|
|
Flux.1, fp6 + fp4 recipe |
|
|
GPT-OSS-20B |
|
|
The Dockerfiles pin every dependency to a fixed commit, and some of them are compiled from source:
Llama2-70B LoRA compiles Transformer Engine and Primus-Turbo, which takes a while. Run
sudo chmod -R 777 .in the implementation folder before building so that all scripts are writable.DLRMv4 compiles FBGEMM-GPU, because there is no public gfx950 build. This takes 30–60 minutes;
--build-arg MAX_JOBS=<n>sets the compile parallelism.Flux.1 runs on every node in the Slurm job, so each node needs the image. Either build it on every node, or push it to a registry that all nodes can reach and set
CONTto that name.
Disclaimers#
The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED “AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, AMD Instinct, EPYC, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved