After being once again frustrated by the CUDA installation experience, I thought that I should give those containers a try. Unfortunately, my computer did not boot anymore after following the installation instructions for the NVIDIA container toolkit as outlined on the NVIDIA website. Reinstalling everything and following the instructions from some random blog post made it work, but I then found that the container with the CUDA version that I needed had been deprecated.

There were other problems, such as the research cluster of my university not having Docker, but that is a different issue.

Containers don't include drivers which is the primary reason for issues.

Containers afair rely on the exact driver version matching between the host system and the container itself.

We were on AWS when we used this so setting up seemed easy enough - AWS gave you the driver, and a matching docker image was easy enough to find.

That's not the case, CUDA containers user space does not have to match the host drivers CUDA capability. The container needs to be the same major version or lower. So a system with a CUDA 13 capable driver should be able to run all previous versions.

For some versions there's even sometimes compat layers built into the container to allow forward version compatibility.