Skip to content

feat: add common version file for dockerfile #1039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
benmalef wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat: add common version file for dockerfile #1039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cc3752c
feat: add common version file for dockerfile
benmalef Oct 23, 2025
1cd60ee
chore: minor changes
benmalef Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 43 additions & 15 deletions Dockerfile-CPU
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,54 @@
FROM ubuntu:22.04

LABEL github="https://github.com/mlcommons/GaNDLF"
LABEL docs="https://mlcommons.github.io/GaNDLF/"
LABEL version=1.0
# ARG SETUPTOOLS_USE_DISTUTILS=local

ARG DEBIAN_FRONTEND=noninteractive
ARG BUILD_TYPE=cpu

# Install jq early to parse JSON
RUN apt-get update && apt-get install -y jq

# Copy and read versions file
COPY versions.json /tmp/versions.json

# Display versions being used
RUN echo "=== Building ${BUILD_TYPE} version with the following versions ===" && \
cat /tmp/versions.json && \
echo "==========================================="

# Install fresh Python and dependencies for build-from-source
RUN apt-get update && apt-get install -y software-properties-common
RUN apt-get install -y git
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt install -y python3.11
RUN apt install -y libpython3.11-dev
RUN apt install -y libpython3.11
# RUN apt install -y python-distutils-extra
RUN apt install -y python3.11-venv
# RUN apt install -y python-setuptools
RUN apt install -y python3-pip
RUN apt-get install -y software-properties-common git

# Install Python using version from JSON
RUN PYTHON_VERSION=$(jq -r '.python' /tmp/versions.json) && \
echo "Installing Python ${PYTHON_VERSION}" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt install -y python${PYTHON_VERSION} \
libpython${PYTHON_VERSION}-dev \
libpython${PYTHON_VERSION} \
python${PYTHON_VERSION}-venv \
python3-pip

# Install additional dependencies
RUN apt-get install -y libjpeg8-dev zlib1g-dev libffi-dev libgl1
# fix pip version because of weird PyYAML issue

# Fix pip version because of weird PyYAML issue
RUN python3.11 -m pip install --upgrade pip
RUN python3.11 -m pip install uv
# EXPLICITLY install cpu versions of torch/torchvision (not all versions have +cpu modes on PyPI...)
RUN uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cpu --system

# EXPLICITLY install cpu versions of torch/torchvision using versions from JSON
RUN PYTORCH_VERSION=$(jq -r '.pytorch.version' /tmp/versions.json) && \
TORCHVISION_VERSION=$(jq -r '.pytorch.torchvision' /tmp/versions.json) && \
TORCHAUDIO_VERSION=$(jq -r '.pytorch.torchaudio' /tmp/versions.json) && \
PYTORCH_INDEX=$(jq -r '.pytorch.index.cpu' /tmp/versions.json) && \
echo "Installing PyTorch ${PYTORCH_VERSION} (${PYTORCH_INDEX}), TorchVision ${TORCHVISION_VERSION}, TorchAudio ${TORCHAUDIO_VERSION}" && \
uv pip install torch==${PYTORCH_VERSION} \
torchvision==${TORCHVISION_VERSION} \
torchaudio==${TORCHAUDIO_VERSION} \
--index-url https://download.pytorch.org/whl/${PYTORCH_INDEX} --system

RUN uv pip install openvino-dev opencv-python-headless --system

# Do some dependency installation separately here to make layer caching more efficient
Expand All @@ -32,6 +59,7 @@ RUN python3.11 -c "from setup import requirements; file = open('requirements.txt
COPY . /GaNDLF
WORKDIR /GaNDLF
RUN uv pip install -e . --system

# Entrypoint forces all commands given via "docker run" to go through python, CMD forces the default entrypoint script argument to be gandlf run
# If a user calls "docker run gandlf:[tag] anonymize", it will resolve to running "gandlf anonymize" instead.
# CMD is inherently overridden by args to "docker run", entrypoint is constant.
Expand All @@ -48,4 +76,4 @@ CMD run
#USER nonroot

# Prepare the container for possible model embedding later.
RUN mkdir /embedded_model
RUN mkdir /embedded_model
61 changes: 44 additions & 17 deletions Dockerfile-CUDA11.8
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,56 @@
FROM nvidia/cuda:11.8.0-devel-ubuntu22.04

LABEL github="https://github.com/mlcommons/GaNDLF"
LABEL docs="https://mlcommons.github.io/GaNDLF/"
LABEL version=1.0
# ARG SETUPTOOLS_USE_DISTUTILS=local

# Install instructions for NVIDIA Container Toolkit allowing you to use the host's GPU: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
# Note that to do this on a Windows host you need experimental feature "CUDA on WSL" -- not yet stable.
ENV DEBIAN_FRONTEND=noninteractive

# Explicitly install python3.11 (this uses 11.1 for now, as PyTorch LTS 1.8.2 is built against it)
RUN apt-get update && apt-get install -y software-properties-common
RUN apt-get install -y git
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt install -y python3.11
RUN apt install -y libpython3.11-dev
RUN apt install -y libpython3.11
# RUN apt install -y python-distutils-extra
RUN apt install -y python3.11-venv
# RUN apt install -y python-setuptools
RUN apt install -y python3-pip
ENV DEBIAN_FRONTEND=noninteractive
ARG BUILD_TYPE=cuda118

# Install jq early to parse JSON
RUN apt-get update && apt-get install -y jq

# Copy and read versions file
COPY versions.json /tmp/versions.json

# Display versions being used
RUN echo "=== Building ${BUILD_TYPE} version with the following versions ===" && \
cat /tmp/versions.json && \
echo "==========================================="

# Explicitly install python3.11
RUN apt-get install -y software-properties-common git

# Install Python using version from JSON
RUN PYTHON_VERSION=$(jq -r '.python' /tmp/versions.json) && \
echo "Installing Python ${PYTHON_VERSION}" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt install -y python${PYTHON_VERSION} \
libpython${PYTHON_VERSION}-dev \
libpython${PYTHON_VERSION} \
python${PYTHON_VERSION}-venv \
python3-pip

# Install additional dependencies
RUN apt-get install -y libjpeg8-dev zlib1g-dev libffi-dev libgl1
# fix pip version because of weird PyYAML issue

# Fix pip version because of weird PyYAML issue
RUN python3.11 -m pip install --upgrade pip
RUN python3.11 -m pip install uv
RUN uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu118 --system

# Install PyTorch packages with CUDA 11.8 support using versions from JSON
RUN PYTORCH_VERSION=$(jq -r '.pytorch.version' /tmp/versions.json) && \
TORCHVISION_VERSION=$(jq -r '.pytorch.torchvision' /tmp/versions.json) && \
TORCHAUDIO_VERSION=$(jq -r '.pytorch.torchaudio' /tmp/versions.json) && \
PYTORCH_INDEX=$(jq -r '.pytorch.index.cuda118' /tmp/versions.json) && \
echo "Installing PyTorch ${PYTORCH_VERSION} (${PYTORCH_INDEX}), TorchVision ${TORCHVISION_VERSION}, TorchAudio ${TORCHAUDIO_VERSION}" && \
uv pip install torch==${PYTORCH_VERSION} \
torchvision==${TORCHVISION_VERSION} \
torchaudio==${TORCHAUDIO_VERSION} \
--index-url https://download.pytorch.org/whl/${PYTORCH_INDEX} --system

RUN uv pip install openvino-dev opencv-python-headless --system

# Do some dependency installation separately here to make layer caching more efficient
Expand Down Expand Up @@ -51,4 +78,4 @@ CMD run
#USER nonroot

# Prepare the container for possible model embedding later.
RUN mkdir /embedded_model
RUN mkdir /embedded_model
61 changes: 44 additions & 17 deletions Dockerfile-CUDA12.6
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,56 @@
FROM nvidia/cuda:12.6.3-cudnn-devel-ubuntu22.04

LABEL github="https://github.com/mlcommons/GaNDLF"
LABEL docs="https://mlcommons.github.io/GaNDLF/"
LABEL version=1.0
# ARG SETUPTOOLS_USE_DISTUTILS=local

# Install instructions for NVIDIA Container Toolkit allowing you to use the host's GPU: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
# Note that to do this on a Windows host you need experimental feature "CUDA on WSL" -- not yet stable.
ENV DEBIAN_FRONTEND=noninteractive

# Explicitly install python3.11 (this uses 11.1 for now, as PyTorch LTS 1.8.2 is built against it)
RUN apt-get update && apt-get install -y software-properties-common
RUN apt-get install -y git
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt install -y python3.11
RUN apt install -y libpython3.11-dev
RUN apt install -y libpython3.11
# RUN apt install -y python-distutils-extra
RUN apt install -y python3.11-venv
# RUN apt install -y python-setuptools
RUN apt install -y python3-pip
ENV DEBIAN_FRONTEND=noninteractive
ARG BUILD_TYPE=cuda126

# Install jq early to parse JSON
RUN apt-get update && apt-get install -y jq

# Copy and read versions file
COPY versions.json /tmp/versions.json

# Display versions being used
RUN echo "=== Building ${BUILD_TYPE} version with the following versions ===" && \
cat /tmp/versions.json && \
echo "==========================================="

# Explicitly install python3.11
RUN apt-get install -y software-properties-common git

# Install Python using version from JSON
RUN PYTHON_VERSION=$(jq -r '.python' /tmp/versions.json) && \
echo "Installing Python ${PYTHON_VERSION}" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt install -y python${PYTHON_VERSION} \
libpython${PYTHON_VERSION}-dev \
libpython${PYTHON_VERSION} \
python${PYTHON_VERSION}-venv \
python3-pip

# Install additional dependencies
RUN apt-get install -y libjpeg8-dev zlib1g-dev libffi-dev libgl1
# fix pip version because of weird PyYAML issue

# Fix pip version because of weird PyYAML issue
RUN python3.11 -m pip install --upgrade pip
RUN python3.11 -m pip install uv
RUN uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126 --system

# Install PyTorch packages with CUDA 12.6 support using versions from JSON
RUN PYTORCH_VERSION=$(jq -r '.pytorch.version' /tmp/versions.json) && \
TORCHVISION_VERSION=$(jq -r '.pytorch.torchvision' /tmp/versions.json) && \
TORCHAUDIO_VERSION=$(jq -r '.pytorch.torchaudio' /tmp/versions.json) && \
PYTORCH_INDEX=$(jq -r '.pytorch.index.cuda126' /tmp/versions.json) && \
echo "Installing PyTorch ${PYTORCH_VERSION} (${PYTORCH_INDEX}), TorchVision ${TORCHVISION_VERSION}, TorchAudio ${TORCHAUDIO_VERSION}" && \
uv pip install torch==${PYTORCH_VERSION} \
torchvision==${TORCHVISION_VERSION} \
torchaudio==${TORCHAUDIO_VERSION} \
--index-url https://download.pytorch.org/whl/${PYTORCH_INDEX} --system

RUN uv pip install openvino-dev opencv-python-headless --system

# Do some dependency installation separately here to make layer caching more efficient
Expand Down Expand Up @@ -51,4 +78,4 @@ CMD run
#USER nonroot

# Prepare the container for possible model embedding later.
RUN mkdir /embedded_model
RUN mkdir /embedded_model
59 changes: 43 additions & 16 deletions Dockerfile-ROCm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,55 @@
FROM rocm/pytorch:rocm6.3.4_ubuntu22.04_py3.10_pytorch_release_2.4.0

LABEL github="https://github.com/mlcommons/GaNDLF"
LABEL docs="https://mlcommons.github.io/GaNDLF/"
LABEL version=1.0
# ARG SETUPTOOLS_USE_DISTUTILS=local

# Quick start instructions on using Docker with ROCm: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master/quick-start.md

ARG BUILD_TYPE=rocm

# Install jq early to parse JSON
RUN apt-get update && apt-get install -y jq

# Copy and read versions file
COPY versions.json /tmp/versions.json

# Display versions being used
RUN echo "=== Building ${BUILD_TYPE} version with the following versions ===" && \
cat /tmp/versions.json && \
echo "==========================================="

# The base image contains ROCm, python 3.9 and pytorch already, no need to install those
RUN apt-get update && apt-get install -y software-properties-common
RUN apt-get install -y git
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt install -y python3.11
RUN apt install -y libpython3.11-dev
RUN apt install -y libpython3.11
# RUN apt install -y python-distutils-extra
RUN apt install -y python3.11-venv
# RUN apt install -y python-setuptools
RUN apt install -y python3-pip
RUN apt-get install -y python3.11 python3-pip libjpeg8-dev zlib1g-dev python3-dev libpython3.11-dev libffi-dev libgl1
# fix pip version because of weird PyYAML issue
RUN apt-get install -y software-properties-common git

# Install Python using version from JSON
RUN PYTHON_VERSION=$(jq -r '.python' /tmp/versions.json) && \
echo "Installing Python ${PYTHON_VERSION}" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt install -y python${PYTHON_VERSION} \
libpython${PYTHON_VERSION}-dev \
libpython${PYTHON_VERSION} \
python${PYTHON_VERSION}-venv \
python3-pip

# Install additional dependencies
RUN apt-get install -y libjpeg8-dev zlib1g-dev python3-dev libffi-dev libgl1

# Fix pip version because of weird PyYAML issue
RUN python3.11 -m pip install --upgrade pip
RUN python3.11 -m pip install uv
RUN uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/rocm6.3 --system

# Install PyTorch packages with ROCm support using versions from JSON
RUN PYTORCH_VERSION=$(jq -r '.pytorch.version' /tmp/versions.json) && \
TORCHVISION_VERSION=$(jq -r '.pytorch.torchvision' /tmp/versions.json) && \
TORCHAUDIO_VERSION=$(jq -r '.pytorch.torchaudio' /tmp/versions.json) && \
PYTORCH_INDEX=$(jq -r '.pytorch.index.rocm' /tmp/versions.json) && \
echo "Installing PyTorch ${PYTORCH_VERSION} (${PYTORCH_INDEX}), TorchVision ${TORCHVISION_VERSION}, TorchAudio ${TORCHAUDIO_VERSION}" && \
uv pip install torch==${PYTORCH_VERSION} \
torchvision==${TORCHVISION_VERSION} \
torchaudio==${TORCHAUDIO_VERSION} \
--index-url https://download.pytorch.org/whl/${PYTORCH_INDEX} --system

RUN uv pip install openvino-dev opencv-python-headless --system

# Do some dependency installation separately here to make layer caching more efficient
Expand All @@ -39,7 +67,6 @@ RUN uv pip install -e . --system
ENTRYPOINT gandlf
CMD run


# The below force the container commands to run as a nonroot user with UID > 10000.
# This greatly reduces UID collision probability between container and host, helping prevent privilege escalation attacks.
# As a side benefit this also decreases the likelihood that users on a cluster won't be able to access their files.
Expand All @@ -50,4 +77,4 @@ CMD run
#USER nonroot

# Prepare the container for possible model embedding later.
RUN mkdir /embedded_model
RUN mkdir /embedded_model
18 changes: 18 additions & 0 deletions versions.json
View file
Open in desktop
Copy link
Collaborator

@sarthakpati sarthakpati Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change the name to versions_docker.json so that the purpose of the file is clear?

Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"python": "3.11",
"pytorch": {
"version": "2.7.1",
Copy link
Collaborator

@sarthakpati sarthakpati Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to pick up the pytorch version from setup.py so that there is a single point of reference?

"torchvision": "0.22.1",
"torchaudio": "2.7.1",
Comment on lines +5 to +6
Copy link
Collaborator

@sarthakpati sarthakpati Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these versions are required.

Copy link
Contributor Author

@benmalef benmalef Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this PR.

Copy link
Collaborator

@sarthakpati sarthakpati Oct 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks. I believe they are coming from the PyTorch install page [ref], where each PT version has an associated vision/audio dependency. Any way we can this more seamless? Would just putting torchvision and torchaudio in the pip command without versions work?

"index": {
"cpu": "cpu",
"cuda118": "cu118",
"cuda126": "cu126",
"rocm": "rocm6.3"
}
},
"dependencies": {
"pip": "latest",
"uv": "latest"
}
}
Loading