Installing ROCm from source with Spack#
In this guide you will learn how Spack makes building ROCm components from source easier and more flexible than other methods. This blog will walk you through installing ROCm from source using the Spack package manager. We will also discuss Spack’s place among other ROCm installation methods, the landscape of ROCm components, and show you how ROCm, as an open-source software platform, allows developers to streamline software stacks for their applications.
What is Spack?#
From the Spack website:
Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy.
Spack began as a from-source package manager, and while the team at Spack are working to provide binaries for all Spack packages, this is still the package manager’s greatest strength.
A from-source package manager is just that—a package manager that builds its packages from source code (when available). Compare this to a traditional package manager (e.g., dpkg/apt for Debian-based distros, or yum/dnf for RHEL-based distros) which installs pre-compiled binaries for each package. Put (extremely) simply, you can think of locally compiled binaries as being “fine-tuned” for the particular machine being used. Because Spack allows you to select the compilers and build optimizations when compiling source code, there is potential to achieve greater performance. Additionally, for GPU accelerated tools and libraries like those included in ROCm, building from source allows you to target specific GPU platforms rather than building for all supported architectures. This can result in reduction in storage costs for installing ROCm, which can be beneficial for creating lightweight container images that depend on ROCm.
Why install ROCm from source?#
If you are installing pre-compiled binaries on a supported operating system (e.g., using apt to install ROCm on Ubuntu 24.04), ROCm and its components come pre-built for any ROCm-supported GPU. These builds are ideal for most Linux package manager installs, so that everyday users need not worry about specific gfx versions.
When installed through a traditional package manager, every component is built against every (supported) gfx version. For example, rocBLAS builds against fourteen different gfx versions (including xnack variants). For example, if you are developing a ROCm-dependent application for a cluster of AMD Instinct™ MI300X Accelerators, you would only need ROCm components and kernels built against gfx942.
There are a few different methods by which you can install ROCm from source, such as make and TheRock, AMD’s lightweight open source build system for HIP and ROCm. The advantage of Spack is that it provides incredible ease-of-use. With each release of ROCm, AMD has consistently kept the corresponding Spack packages up to date. Continuing with our MI300X example, we can just spack install hipblas to install hipBLAS. (We can further influence the specifics of this build process, as we will see below.)
Install ROCm accelerated packages with Spack#
The ROCm Documentation provides a detailed guide on using Spack.
Getting started with Spack is quite easy. To install Spack, simply clone the repository and source setup-env.sh:
git clone https://github.com/spack/spack.git ~/spack/
source ~/spack/share/spack/setup-env.sh
You must also have C, C++, and Fortran compilers installed. To help Spack find your compilers, you can run the following
spack compiler find
To install hipBLAS targeting CDNA3 architectures, you can start by previewing what Spack will install by using spack spec
spack spec hipblas amdgpu_target=gfx942
Output of spack spec hipblas amdgpu_target=gfx942
- [email protected]~asan~cuda~ipo+rocm amdgpu_target=gfx942 build_system=cmake build_type=Release generator=make patches=8d71578,b05b34b arch=linux-rocky9-zen4
- ^[email protected]~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-rocky9-zen4
- ^[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-zen4
- ^ca-certificates-mozilla@2025-02-25 build_system=generic arch=linux-rocky9-zen4
- ^[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-zen4
- ^[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
[e] ^[email protected]~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages='c,c++,fortran' arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
[e] ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~guile build_system=generic arch=linux-rocky9-zen4
- ^[email protected]~asan~cuda~ipo+rocm build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected] build_system=bundle arch=linux-rocky9-zen4
- ^[email protected]+glx+llvm+opengl~opengles+osmesa~strip build_system=meson buildtype=release default_library=shared arch=linux-rocky9-zen4
- ^[email protected]~color build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+lex~nls build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs~pic+tests+weak_backtrace~xz~zlib build_system=autotools components=none libs=shared,static arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+clang~cuda~flang+gold~ipo+libomptarget~libomptarget_debug~link_llvm_dylib+lld+lldb+llvm_dylib+lua~mlir+polly~python~split_dwarf~z3~zstd build_system=cmake build_type=Release compiler-rt=runtime generator=ninja libcxx=runtime libunwind=runtime openmp=runtime shlib_symbol_version=none targets=all version_suffix=none arch=linux-rocky9-zen4
- ^[email protected]~debuginfod~gas+gold~gprofng+headers~interwork+ld~libiberty~lto~nls~pgo+plugins build_system=autotools compress_debug_sections=zlib libs=shared,static arch=linux-rocky9-zen4
- ^[email protected]~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rocky9-zen4
- ^[email protected]+shared build_system=makefile fetcher=curl arch=linux-rocky9-zen4
- ^[email protected] build_system=makefile patches=881d2ed,f6f6236 arch=linux-rocky9-zen4
- ^[email protected] build_system=perl arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~jit+multibyte+pic build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make patches=c10b010 arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo build_system=cmake build_type=Release generator=make patches=16e0e2b arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools patches=440b954 arch=linux-rocky9-zen4
- ^[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rocky9-zen4
- ^[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-zen4
- ^[email protected]~debug~pic+shared build_system=generic arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=perl arch=linux-rocky9-zen4
- ^[email protected] build_system=perl arch=linux-rocky9-zen4
- ^[email protected] build_system=perl arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make patches=fc2f3cd arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan+image~ipo+shared build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~debuginfod+exeprefix+nls build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools zip=pigz arch=linux-rocky9-zen4
- ^[email protected] build_system=makefile arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools libs=shared,static arch=linux-rocky9-zen4
- ^[email protected]~pic build_system=autotools libs=shared,static arch=linux-rocky9-zen4
- ^[email protected]+programs build_system=makefile compression=none libs=shared,static arch=linux-rocky9-zen4
- ^[email protected]~docs~strip build_system=meson buildtype=release default_library=shared arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip patches=0f0b1bd arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=makefile arch=linux-rocky9-zen4
- ^[email protected]~ipo~link_llvm_dylib~llvm_dylib+rocm-device-libs build_system=cmake build_type=Release generator=ninja patches=b4774ca arch=linux-rocky9-zen4
- ^[email protected]~http+pic~python+shared build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+re2c build_system=generic patches=93f4bb3 arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rocky9-zen4
- ^[email protected]+libbsd build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~obsolete_api build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools patches=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rocky9-zen4
- ^[email protected]+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]~gmp~ipo~python build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo+tensile amdgpu_target=gfx942 build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=c58195d arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo+shared build_system=cmake build_type=Release generator=make patches=62be726 arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]+nls build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
- ^[email protected]+libyaml build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected]+toml build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected]+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+gssapi build_system=autotools arch=linux-rocky9-zen4
- ^[email protected]+shared build_system=autotools arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected]~wheel build_system=python_pip arch=linux-rocky9-zen4
- ^[email protected] build_system=generic arch=linux-rocky9-zen4
- ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo+optimal amdgpu_target=gfx942 build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~ipo+pic~shared build_system=cmake build_type=Release cxxstd=11 generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo~test amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
- ^[email protected]~asan~ipo amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rocky9-zen4
To install, again we just use spack install hipblas amdgpu_target=gfx942, this time specifying a gfx version.
Spack environments#
A Spack environment is used to group a set of specs intended for some purpose to be built, rebuilt, and deployed in a coherent fashion. You can think of an environment as a blueprint for a set of related software specifications (specs). It defines:
Which packages to install
How each package is configured
Where everything gets installed
Instead of installing packages one-by-one and manually loading modules, environments let you group specs together for a specific purpose; e.g. for creating virtual machine or Docker images for deployment in a cloud environment targeting a specific AMD GPU architecture.
With a single command, you can concretize, install, or activate the entire environment. In concretization, Spack resolves all dependencies and configuration options across your environment. This step prepares everything for installation in a consistent and reproducible way—far more robust than cobbling together ad-hoc scripts. You may have noticed in our previous spack install hipblas example above, specifying amdgpu_target=gfx942 only applied that configuration to the hipblas package; this variant does not necessarily propagate to its dependencies. To ensure the amdgpu_target option propagates to all packages that have this variant, we can use the packages.all.prefer: ["amdgpu_target=gfx942"] option.
Environments also provide stability. Even if upstream Spack packages change, your environment remains unchanged until you choose to re-concretize. That makes it easy to reproduce builds and maintain consistency over time.
By specifying install paths, environments give you a clear filesystem view of what’s been installed. Under the hood, Spack still uses a single, deduplicated software installation that can be shared across multiple environments—saving space and reducing rebuilds. Spack Environments can have an associated filesystem view, which is a directory with a more traditional structure, e.g. <view>/bin, <view>/lib, <view>/include in which all files of the installed packages in the environment are linked.
Finally, activating an environment loads only the modules you need for that specific workflow. Spack can even generate shell scripts to automate module loading for the environment—keeping your software environment lean, relevant, and predictable.
As an example, we can set up an environment using a yaml file that defines what packages we want to install, how to install them, and where to install them. Below is an example environment file that installs hipblas and all of its dependencies targeting the gfx942 AMD GPU architecture.
spack:
specs:
- [email protected]
concretizer:
unify: true
packages:
all:
prefer:
- "amdgpu_target=gfx942"
view: /opt/rocm
In this example, hipBLAS and all of its dependencies will be installed in a single Spack managed location and the filesystem view of the packages will be available at /opt/rocm. In essence, once this environment is installed, this would provide a hipBLAS installation in much the same way that apt install hipblas would.
To work with environments, suppose this example environment file is stored in $HOME/spack-env/spack.yaml. You can activate this environment and concretize the build with the following command:
spack env activate $HOME/spack-env
spack concretize -f
To install this environment, simply run:
spack install
spack gc -y
The last step here performs “garbage collection”, removing packages that are only build-time dependencies. This reduces image size by removing packages that are not required at runtime and helps lighten up the final installation.
Read more about Spack environments here.
Using Spack environments to create Docker image recipes#
Another major benefit of using a Spack environment is that you can easily containerize it to ensure reproducibility and portability. To do this, you can create a Dockerfile to install just the ROCm packages you care about for the specific architectures you’d like to target. In continuing with our previous example, we can create a Dockerfile by doing the following :
# Navigate to the directory where your spack.yaml file is located
cd $HOME/spack-env
spack containerize > Dockerfile
This creates a two-stage build recipe that starts from Spack managed container images. In the first stage of the build (the “builder” stage), the environment is concretized and installed in the container and build-time dependencies are removed. Following this, all the binary files (executables, shared libraries, and static archives) found under the filesystem view are stripped of their debugging symbols and other non-essential metadata to further reduce the image size. The second build stage builds on a compatible base OS image without Spack installed. The file system view and installation tree are copied from the “builder stage” and the entrypoint is set to a shell that is preconfigured with the Spack view paths defined in the environment.
The full contents of the Dockerfile generated from this example are shown below.
Output of spack containerize > Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-jammy:develop AS builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir -p /opt/spack-environment && \
set -o noclobber \
&& (echo spack: \
&& echo ' specs:' \
&& echo ' - [email protected]' \
&& echo ' concretizer:' \
&& echo ' unify: true' \
&& echo ' packages:' \
&& echo ' all:' \
&& echo ' prefer:' \
&& echo ' - amdgpu_target=gfx942' \
&& echo ' view: /opt/views/view' \
&& echo '' \
&& echo ' config:' \
&& echo ' install_tree: /opt/software') > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
# Strip all the binaries
RUN find -L /opt/views/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . > activate.sh
# Bare OS image to run the installed executables
FROM ubuntu:22.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
# paths.view is a symlink, so copy the parent to avoid dereferencing and duplicating it
COPY --from=builder /opt/views /opt/views
RUN { \
echo '#!/bin/sh' \
&& echo '.' /opt/spack-environment/activate.sh \
&& echo 'exec "$@"'; \
} > /entrypoint.sh \
&& chmod a+x /entrypoint.sh \
&& ln -s /opt/views/view /opt/view
ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "/bin/bash" ]
Learn more about creating containers with Spack.
Using Spack to understand ROCm dependencies#
Understanding ROCm Components#
ROCm is not one piece of software. It is a collection of interconnected, focused components, such the HIP runtime API for heterogenous systems, rocBLAS (AMD’s BLAS implementation for AMD GPUs), hipFORT (the HIP Fortran library), various compilers, and many more application-specific tools and libraries.
ROCm officially supports roughly 70 components, but the exact list of component varies across multiple sources. Here are a few:
What is ROCm? \(^{[1]}\)
ROCm/default.xml\(^{[2]}\)ROCm Documentation for Spack \(^{[1]}\)
ROCm Compatibility Matrix \(^{[1]}\)
Version release notes \(^{[1]}\)
\(^{[1]}\) Pages from ROCm Documentation
\(^{[2]}\) ROCm GitHub repository
The landscape of ROCm components is best laid out by the following graphic (Figure 1, below), taken from the What is ROCm? page from the documentation.
Figure 1: The ROCm software stack
The ROCm stack relies on lower layers of runtimes and compilers which are generally essential for most ROCm components; i.e., these are the “meat and potatoes” of ROCm. Above these layers are Tools, which are quite handy for developers, but are also used by some ROCm components. The ROCm stack becomes particularly discrete at the Library level. E.g., rocJPEG does not depend on hipTensor and vice versa. This motivates the question, “Why should we install all of ROCm?” if we just need a specific component, and drives us to understand specific dependencies within the ROCm stack.
ROCm Component Dependencies#
As a package manager, Spack is also a great system for understanding the dependencies of the ROCm stack. The following graphic (Figure 2, below) has been generated using the outputs of spack spec <package>.
Figure 2: Dependency matrix for ROCm 6.3.2.
Light squares indicate that the package on the Y-axis does not depend on the given package on the X-axis. Dark squares indicate that there is a dependency. E.g., hip depends on aqlprofile, hipcc, hipify-clang, hsa-rocr-dev, llvm-amdgpu, rocm-cmake, rocm-core, rocminfo, and rocprofiler-register.
Note: Package names are presented as they are in Spack. Exact names may differ from apt/dnf packages and GitHub repositories.
This graphic covers all of the major ROCm components that are packaged for Spack. We have also included a version of PyTorch + ROCm for reference. The edge here is the verbosity of spack spec lets us quickly understand the dependencies of tools that use ROCm, not just the interdependencies of ROCm components.
Using Spack to install ROCm#
Beyond even specific gfx versions, your application might not even require all default ROCm components. If you apt depends rocm6.3.2, you’ll notice that the default installation of ROCm includes mivisionx, a set of comprehensive computer vision and machine intelligence libraries, utilities, and applications. Although mivisionx comes packaged with ROCm by default, many ROCm components do not require this component. This underscores the utility of selective component installation via Spack.
Unlike apt or dnf, Spack does NOT maintain a package for a generic rocm/rocm-dev package. Instead, ROCm components are packaged individually. The following spack.yaml approximates an installation of rocm-dev. (Note: none of these packages can be built for specific gfx versions.)
spack.yaml:
spack:
specs:
- amdsmi
- comgr
- hip
- hipcc
- hipify-clang
- aqlprofile
- hsa-rocr-dev
- rocm-openmp-extras
- rocm-cmake
- rocm-core
- rocm-dbgapi
- rocm-debug-agent
- rocm-device-libs
- rocm-gdb
- llvm-amdgpu
- rocm-opencl
- rocm-smi-lib
- rocprofiler-dev
- rocprofiler-register
- rocprofiler-sdk
- roctracer-dev
concretizer:
unify: true
config:
install_tree: $HOME/opt/rocm
view: $HOME/opt/rocm
It’s worth noting that some packages related to ROCm that are available through dnf/apt are not available in Spack, e.g., rocm-utils. Also, some packages such as aqlprofile are not open source, and instead these are installed by grabbing the .deb from repo.radeon.com.
Using Spack to install PyTorch with ROCm#
Beyond ROCm, Spack also provides numerous Python packages, including PyTorch. Below is a spack.yaml that my be used to build PyTorch and its dependencies specifically for MI300X/MI300A.
spack.yaml:
spack:
specs:
- py-torch+rocm
concretizer:
unify: true
packages:
all:
prefer:
- "amdgpu_target=gfx942"
config:
install_tree: $HOME/opt/rocm
view: $HOME/opt/rocm
Drivers#
It is good to remind ourselves that the purpose of the ROCm software stack is to provide developers with the tools to program on (primarily) AMD GPUs. Under the hood, the developer still utilizes drivers to meaningfully interface with their AMD GPU and actually run programs. I.e., Spack and ROCm are used to build applications, drivers are used to run them.
All drivers required for ROCm are fully open source at github.com/ROCm/ROCK-Kernel-Driver.
The Broader ROCm Ecosystem#
So far we have described ~70 components (either as dnf/apt/Spack packages or GitHub repositories) that compose the ROCm software stack. However, if you go to the ROCm GitHub page, you’ll find that the ROCm organization has over 300 repositories. Why so many repositories if ROCm only has ~70 components?
This is the beauty of open-source development. AMD is continuously maintaining ROCm integrations, compatibility, backends, etc. for major GPU computing tools, such as PyTorch, JAX, Tensorflow, transformers, Triton, VLLM, just to list a few. In fact 110 of AMD’s ROCm repositories are forks (as of publication).
Beyond forks, AMD actively creates new tools for ROCm. Some of these developing tools are AITER (AI Tensor Engine for ROCm), rocsift (a C99 debugging API for ROCm), rocRoller (a software library for generating AMDGPU kernels), mxDataGenerator (a library for data generation indifferent floating point formats), and TheRock (a build system for HIP and ROCm).
Summary#
This blog provides a basic introduction to the Spack package manager and how to install ROCm components using Spack. Specific advantages of Spack have been discussed, such as
Easily building ROCm components from source.
Installing Spack packages (such as PyTorch) targeting specific
gfxarchitectures.Using Spack for cleaning up build dependencies.
Utilizing
spack specto understand ROCm dependencies.
This post also highlights the landscape of the ROCm ecosystem and provides an overview of standard and upcoming ROCm components.
Acknowledgments#
Special thanks to Garrett Byrd and Dr. Joe Schoonover at Fluid Numerics for contributing this blog. The ROCm software ecosystem is strengthened by community projects that enable you to use AMD GPUs in new ways. If you have a project you would like to share here, please raise an issue or PR.
Find Fluid Numerics online#
Disclaimers#
Third-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.