Skip to content

Do not use cache in github image build action#15308

Merged
AlanCoding merged 2 commits into
ansible:develfrom
AlanCoding:cache_invalidation
Jun 28, 2024
Merged

Do not use cache in github image build action#15308
AlanCoding merged 2 commits into
ansible:develfrom
AlanCoding:cache_invalidation

Conversation

@AlanCoding
Copy link
Copy Markdown
Member

@AlanCoding AlanCoding commented Jun 27, 2024

SUMMARY

I create this due to struggling with the action:

https://github.com/ansible/awx/actions/workflows/devel_images.yml

It will not pick up a new DAB version.

2024-06-27T13:43:16.1730206Z #34 [linux/amd64 builder  8/10] RUN cd /tmp && make requirements_awx
2024-06-27T13:43:16.1730918Z #34 CACHED

When it does this, it produces an image that uses a DAB git hash 3 commits behind the latest. Clearly, because it never installed requirements. You frequently see these timings:

Screenshot from 2024-06-27 12-02-13

It's probable that requirements were not installed if it took less than 1 minute.

I don't want to re-install requirements on ever PR, which creates the central challenge here.

Either we use --cache-from or we use --no-cache. It is hard to do this in make language. This is my best shot.

ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • API

@AlanCoding
Copy link
Copy Markdown
Member Author

AlanCoding commented Jun 27, 2024

Showing some testing:

(awx) alancoding@fedora:~/repos/awx$ DOCKER_CACHE=--no-cache make --just-print awx-kube-dev-build
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
    -e dockerfile_name=Dockerfile.kube-dev \
    -e kube_dev=True \
    -e template_dest=_build_kube_dev \
    -e receptor_image=quay.io/ansible/receptor:devel
DOCKER_BUILDKIT=1 docker build -f Dockerfile.kube-dev \
    --build-arg BUILDKIT_INLINE_CACHE=1 \
     --no-cache \
    -t ghcr.io/ansible/awx_kube_devel:cache_invalidation .
(awx) alancoding@fedora:~/repos/awx$ make --just-print awx-kube-dev-build
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
    -e dockerfile_name=Dockerfile.kube-dev \
    -e kube_dev=True \
    -e template_dest=_build_kube_dev \
    -e receptor_image=quay.io/ansible/receptor:devel
DOCKER_BUILDKIT=1 docker build -f Dockerfile.kube-dev \
    --build-arg BUILDKIT_INLINE_CACHE=1 \
     --cache-from=ghcr.io/ansible/awx_kube_devel:cache_invalidation \
    -t ghcr.io/ansible/awx_kube_devel:cache_invalidation .
(awx) alancoding@fedora:~/repos/awx$ 
(awx) alancoding@fedora:~/repos/awx$ DOCKER_CACHE=--no-cache make --just-print docker-compose-build
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
	-e dockerfile_name=Dockerfile.dev \
	-e build_dev=True \
	-e receptor_image=quay.io/ansible/receptor:devel
DOCKER_BUILDKIT=1 docker build \
	-f Dockerfile.dev \
	-t ghcr.io/ansible/awx_devel:cache_invalidation \
	--build-arg BUILDKIT_INLINE_CACHE=1 \
	--no-cache .
(awx) alancoding@fedora:~/repos/awx$ make --just-print docker-compose-build
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
	-e dockerfile_name=Dockerfile.dev \
	-e build_dev=True \
	-e receptor_image=quay.io/ansible/receptor:devel
DOCKER_BUILDKIT=1 docker build \
	-f Dockerfile.dev \
	-t ghcr.io/ansible/awx_devel:cache_invalidation \
	--build-arg BUILDKIT_INLINE_CACHE=1 \
	--cache-from=ghcr.io/ansible/awx_devel:cache_invalidation .
(awx) alancoding@fedora:~/repos/awx$ 
(awx) alancoding@fedora:~/repos/awx$ DOCKER_CACHE=--no-cache make --just-print awx-kube-buildx
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
	-e receptor_image=quay.io/ansible/receptor:devel \
	-e headless=no
docker buildx create --name awx-kube-buildx
docker buildx use awx-kube-buildx
docker buildx build \
	--push \
	--build-arg VERSION=24.0.1.dev195+g763932a9dc \
	--build-arg SETUPTOOLS_SCM_PRETEND_VERSION=24.0.1.dev195+g763932a9dc \
	--build-arg HEADLESS=no \
	--platform=linux/amd64,linux/arm64   \
	--no-cache \
	--tag ghcr.io/ansible/awx:cache_invalidation \
	-f Dockerfile .
docker buildx rm awx-kube-buildx
(awx) alancoding@fedora:~/repos/awx$ make --just-print awx-kube-buildx
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \
	-e receptor_image=quay.io/ansible/receptor:devel \
	-e headless=no
docker buildx create --name awx-kube-buildx
docker buildx use awx-kube-buildx
docker buildx build \
	--push \
	--build-arg VERSION=24.0.1.dev195+g763932a9dc \
	--build-arg SETUPTOOLS_SCM_PRETEND_VERSION=24.0.1.dev195+g763932a9dc \
	--build-arg HEADLESS=no \
	--platform=linux/amd64,linux/arm64   \
	--cache-from=ghcr.io/ansible/awx:cache_invalidation \
	--tag ghcr.io/ansible/awx:cache_invalidation \
	-f Dockerfile .
docker buildx rm awx-kube-buildx

@AlanCoding
Copy link
Copy Markdown
Member Author

AlanCoding commented Jun 27, 2024

I merged stuff today, for a fact, and it ran the job to build the image for a fact.

$ docker pull ghcr.io/ansible/awx_devel:devel
devel: Pulling from ansible/awx_devel
Digest: sha256:e3cab6e2ce737854b66f8440cf146f1fc77c9a8f0d413437ef3ba5d3336dde61
Status: Image is up to date for ghcr.io/ansible/awx_devel:devel
ghcr.io/ansible/awx_devel:devel

Yet I still see:

$ docker images
REPOSITORY                         TAG          IMAGE ID       CREATED         SIZE
ghcr.io/ansible/awx_devel          devel        7e5cd32a1cfa   25 hours ago    2.1GB

It definitely ran the command to push the image. The problem is that the image it pushes has the same hash as the image before. So no change, that's why my image shows it modified 25 hours ago.

@AlanCoding
Copy link
Copy Markdown
Member Author

AlanCoding commented Jun 28, 2024

Confirming from check output that this does not undo the cache for checks:

2024-06-27T19:13:51.2618431Z DOCKER_BUILDKIT=1 docker build \
2024-06-27T19:13:51.2621670Z 	-f Dockerfile.dev \
2024-06-27T19:13:51.2622383Z 	-t ghcr.io/ansible/awx_devel:devel \
2024-06-27T19:13:51.2623158Z 	--build-arg BUILDKIT_INLINE_CACHE=1 \
2024-06-27T19:13:51.2623840Z 	--cache-from=ghcr.io/ansible/awx_devel:devel .
2024-06-27T19:13:51.7924967Z #0 building with "default" instance using docker driver

@AlanCoding AlanCoding merged commit a70b0c1 into ansible:devel Jun 28, 2024
@AlanCoding
Copy link
Copy Markdown
Member Author

AlanCoding commented Jun 28, 2024

Now I'm closely watching the post-merge action:

https://github.com/ansible/awx/actions/runs/9713929243/job/26811829563

docker buildx use docker-compose-buildx
docker buildx build \
	--push \
	--build-arg BUILDKIT_INLINE_CACHE=1 \
	--no-cache \
	--platform=linux/amd64,linux/arm64   \
	--tag ghcr.io/ansible/awx_devel:devel \
	-f Dockerfile.dev .

That line initially makes it look like it's working.

@AlanCoding
Copy link
Copy Markdown
Member Author

AlanCoding commented Jun 28, 2024

No longer caching:

Screenshot from 2024-06-28 11-10-27

$ docker images
REPOSITORY                         TAG          IMAGE ID       CREATED             SIZE
ghcr.io/ansible/awx_devel          devel        313185c32383   About an hour ago   2.1GB

@AlanCoding AlanCoding mentioned this pull request Jun 28, 2024
Merged
djyasin pushed a commit to djyasin/awx that referenced this pull request Sep 16, 2024
@AlanCoding @djyasin
Do not use cache in github image build action (ansible#15308)
2718490 
* Do not use cache in actual image build action

* Add cache args to kube prod builds
djyasin pushed a commit to djyasin/awx that referenced this pull request Nov 11, 2024
@AlanCoding @djyasin
Do not use cache in github image build action (ansible#15308)
7d20b92 
* Do not use cache in actual image build action

* Add cache args to kube prod builds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrismeyersfsu chrismeyersfsu chrismeyersfsu approved these changes

@TheRealHaoLiu TheRealHaoLiu TheRealHaoLiu approved these changes

@shanemcd shanemcd Awaiting requested review from shanemcd

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlanCoding @chrismeyersfsu @TheRealHaoLiu