Reduce CI go module downloads, add make targets by silverwind · Pull Request #18708 · go-gitea/gitea

silverwind · 2022-02-10T17:15:48Z

The CI currently downloads all go modules in each pipeline step because go modules reside outside the project directory. Fix this by introducing a volume for the /go directory [1] so modules are only downloaded once per pipeline using a new deps-backend make target.

For completeness, I also included new deps and deps-frontend targets and the frontend one is also triggered explicitly on CI where needed.

[1] https://docs.drone.io/pipeline/kubernetes/examples/language/golang/#dependencies

The CI currently downloads all go modules in each pipeline step because go modules reside outside the project directory. Fix this by introducing a volume for the `/go` directory [1] so modules are only downloaded once per pipeline using a new `deps-backend` make target. For completeness, I also included new `deps` and `deps-frontend` targets and the frontend one is also triggered explicitly on CI where needed. [1] https://docs.drone.io/pipeline/kubernetes/examples/language/golang/#dependencies

silverwind · 2022-02-10T17:22:49Z

Seems there is a concurrency issue with the lint steps which all try to install golangci-lint in parallel into the shared volume, resulting in golangci-lint: Text file busy. Probably need to extract tool installation to a separate step or include it in deps-backend so it only runs once.

techknowlogick · 2022-02-10T17:54:51Z

@silverwind would "privileged" mode need to be enabled for this? If so, I can sign this yaml

silverwind · 2022-02-10T22:13:48Z

Don't think it need privileges.

silverwind · 2022-02-10T22:30:09Z

Few more downloads to eliminate, on average, this saves about 15 seconds per step that previously downloaded modules. Failures on the database services are unrelated to the PR.

techknowlogick · 2022-02-10T23:36:58Z

Awesome, thanks :)

I've restarted the build

silverwind · 2022-02-11T00:30:17Z

Should be ready for review.

There is one unrelated change included here which may speed up golangci-lint a bit by skipping over certain directories (thought it will not work for all linters, see golangci/golangci-lint#913.

silverwind · 2022-02-11T10:13:36Z

@lunny I've gone through all pull statements and fixed them so they always are used on the first occurence of an image in a pipeline and the order in yaml is now image followed by pull which makes more sense to me.

zeripath · 2022-02-12T07:06:36Z

make lgtm work

zeripath · 2022-02-12T07:09:11Z

I think we might want to consider backporting this to 1.16.

silverwind · 2022-02-12T09:08:40Z

Right, there's no reason not to.

The CI currently downloads all go modules in each pipeline step because go modules reside outside the project directory. Fix this by introducing a volume for the `/go` directory [1] so modules are only downloaded once per pipeline using a new `deps-backend` make target. For completeness, I also included new `deps` and `deps-frontend` targets and the frontend one is also triggered explicitly on CI where needed. [1] https://docs.drone.io/pipeline/kubernetes/examples/language/golang/#dependencies

* giteaofficial/main: Send mail to issue/pr assignee/reviewer also when OnMention is set (go-gitea#18707) Reduce CI go module downloads, add make targets (go-gitea#18708) Add number in queue status to monitor page (go-gitea#18712) Fix source code line highlighting (go-gitea#18729) Fix forked repositories missed tags (go-gitea#18719) [skip ci] Updated translations via Crowdin Fix release typo (go-gitea#18728) Display template path of current page in dev mode (go-gitea#18717) Separate the details links of commit-statuses in headers (go-gitea#18661) Add LDAP group sync to Teams, fixes go-gitea#1395 (go-gitea#16299) Change git.cmd to RunWithContext (go-gitea#18693)

…) (#18741) Backport #18708 Backport #18475 Backport #18443 The CI currently downloads all go modules in each pipeline step because go modules reside outside the project directory. Fix this by introducing a volume for the `/go` directory [1] so modules are only downloaded once per pipeline using a new `deps-backend` make target. For completeness, I also included new `deps` and `deps-frontend` targets and the frontend one is also triggered explicitly on CI where needed. [1] https://docs.drone.io/pipeline/kubernetes/examples/language/golang/#dependencies * Also backports #18475 and #18443 so that is was able to merge cleanly. Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>

* 'main' of https://github.com/go-gitea/gitea: (87 commits) Fix template bug of LFS lock (go-gitea#18784) Various Mermaid improvements (go-gitea#18776) [skip ci] Updated translations via Crowdin Fix display time of milestones (go-gitea#18753) [skip ci] Updated translations via Crowdin Prevent dangling GetAttribute calls (go-gitea#18754) Add example to render html files (go-gitea#18736) Fix a broken link in `commits_list_small.tmpl` (go-gitea#18763) Fix broken cancel button link on patch page (go-gitea#18718) Ignore the migrate if u2f_registration is not exist (go-gitea#18760) [skip ci] Updated translations via Crowdin Increase the size of the webauthn_credential credential_id field (go-gitea#18739) Fix isempty detection of git repository (go-gitea#18746) [skip ci] Updated translations via Crowdin Send mail to issue/pr assignee/reviewer also when OnMention is set (go-gitea#18707) Reduce CI go module downloads, add make targets (go-gitea#18708) Add number in queue status to monitor page (go-gitea#18712) Fix source code line highlighting (go-gitea#18729) Fix forked repositories missed tags (go-gitea#18719) [skip ci] Updated translations via Crowdin ...

The CI currently downloads all go modules in each pipeline step because go modules reside outside the project directory. Fix this by introducing a volume for the `/go` directory [1] so modules are only downloaded once per pipeline using a new `deps-backend` make target. For completeness, I also included new `deps` and `deps-frontend` targets and the frontend one is also triggered explicitly on CI where needed. [1] https://docs.drone.io/pipeline/kubernetes/examples/language/golang/#dependencies

silverwind added the topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile label Feb 10, 2022

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 10, 2022

silverwind changed the title ~~reduce CI go module downloads, add make targets~~ Reduce CI go module downloads, add make targets Feb 10, 2022

remove volumes from lint tasks to prevent concurrency issue

ae5ab05

silverwind added the pr/wip This PR is not ready for review label Feb 10, 2022

silverwind added 4 commits February 11, 2022 01:15

add few more dependencies to avoid double downloads

0dc1004

run checks-backend in parallel with lint

a9760a6

skip node_modules with golangci

114412c

add some more skips-dirs

6a77a0a

silverwind removed the pr/wip This PR is not ready for review label Feb 11, 2022

silverwind added the skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. label Feb 11, 2022

lunny reviewed Feb 11, 2022

View reviewed changes

Comment thread .drone.yml

fix and re-arrange pull options

d67fc2a

restart ci

bbd72c9

techknowlogick approved these changes Feb 11, 2022

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 11, 2022

Merge branch 'main' into modules-cache-ci

8bed4d4

lafriks approved these changes Feb 11, 2022

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 11, 2022

Merge branch 'main' into modules-cache-ci

ec2014a

techknowlogick and others added 3 commits February 11, 2022 22:35

Merge branch 'main' into modules-cache-ci

7fc5b03

Merge branch 'main' into modules-cache-ci

d5f591e

Merge branch 'main' into modules-cache-ci

08c01b5

zeripath merged commit 2ddff74 into go-gitea:main Feb 12, 2022

silverwind deleted the modules-cache-ci branch February 12, 2022 09:08

lunny added the backport/v1.16 label Feb 12, 2022

silverwind mentioned this pull request Feb 12, 2022

Reduce CI go module downloads, add make targets (#18708, #18475, #18443) #18741

Merged

zeripath added the backport/done All backports for this PR have been created label Feb 12, 2022

zeripath added this to the 1.17.0 milestone Feb 12, 2022

zeripath mentioned this pull request Apr 19, 2022

golangci-lint should ignore node_modules #18803

Closed

go-gitea locked and limited conversation to collaborators Apr 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reduce CI go module downloads, add make targets#18708

Reduce CI go module downloads, add make targets#18708
zeripath merged 13 commits into
go-gitea:mainfrom
silverwind:modules-cache-ci

silverwind commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022 •

edited

Loading

Uh oh!

techknowlogick commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022 •

edited

Loading

Uh oh!

techknowlogick commented Feb 10, 2022

Uh oh!

silverwind commented Feb 11, 2022

Uh oh!

Uh oh!

silverwind commented Feb 11, 2022

Uh oh!

zeripath commented Feb 12, 2022

Uh oh!

zeripath commented Feb 12, 2022

Uh oh!

silverwind commented Feb 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

silverwind commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

techknowlogick commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022

Uh oh!

silverwind commented Feb 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

techknowlogick commented Feb 10, 2022

Uh oh!

silverwind commented Feb 11, 2022

Uh oh!

Uh oh!

silverwind commented Feb 11, 2022

Uh oh!

zeripath commented Feb 12, 2022

Uh oh!

zeripath commented Feb 12, 2022

Uh oh!

silverwind commented Feb 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

silverwind commented Feb 10, 2022 •

edited

Loading

silverwind commented Feb 10, 2022 •

edited

Loading