Repository navigation
ci: some housekeeping and modernizations - #2445
Open
pks-gitlab wants to merge 17 commits into
Open
pks-gitlab wants to merge 17 commits into
pks-gitlab wants to merge 17 commits into
Conversation
We only link curl into a subset of our subcommands. Consequently, as
both "http.c" and "http-walker.c" depend on curl, we don't compile these
into "libgit.a" but instead only link those into the commands that
depend on curl.
In Meson, we wire these dependencies into the target executables by
using the `sources:` keyword. But this has the consequence that we're
recompiling those multiple several times, once for every different
command they are linked into. In fact, each of these sources is compiled
seven times, which of course has an impact on compilation speed.
Fix this issue by instead linking these into a static library so that
they only need to be compiled once. This gives us an almost 10% speedup
in a clean build:
Benchmark 1: meson compile (version = HEAD~)
Time (mean ± σ): 6.781 s ± 0.052 s [User: 100.775 s, System: 22.954 s]
Range (min … max): 6.709 s … 6.867 s 10 runs
Benchmark 2: meson compile (version = HEAD)
Time (mean ± σ): 6.274 s ± 0.021 s [User: 91.882 s, System: 22.092 s]
Range (min … max): 6.242 s … 6.306 s 10 runs
Summary
meson compile (version = HEAD) ran
1.08 ± 0.01 times faster than meson compile (version = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When running our tests, we expect git-remote-http(1) and a couple of
other binaries to be available to the test suite. In our Makefile, we
achieve this by simply hardlinking the file into place in our source
directory. We cannot easily do that in Meson though because there is no
available command to create such a hardlink.
We could of course create a custom target that uses a script for that,
but that feels quite awkward. Instead, we build the executable several
times, which is of course less efficient. Even worse though, similar as
in the preceding commit, we're building "remote-curl.c" once for each of
these targets, which makes this even more expensive.
Fix this by reusing the already-compiled objects from git-remote-http(1)
so that we only have to perform the linking step several times. This
leads to a mild speedup:
Benchmark 1: meson compile (version = HEAD~)
Time (mean ± σ): 6.250 s ± 0.040 s [User: 90.881 s, System: 21.912 s]
Range (min … max): 6.197 s … 6.344 s 10 runs
Benchmark 2: meson compile (version = HEAD)
Time (mean ± σ): 6.218 s ± 0.029 s [User: 90.633 s, System: 22.022 s]
Range (min … max): 6.166 s … 6.262 s 10 runs
Summary
meson compile (version = HEAD) ran
1.01 ± 0.01 times faster than meson compile (version = HEAD~)
Honestly, a 1% speedup isn't really worth it. But the change makes sense
anyway, as we're doing the same when we build git-receive-pack(1) et al.
So while the speed improvement is negligible, it brings more consistency
into our build instructions.
For the record: I also had a look at using a custom target that
hardlinks the files into place. But the improvement it had on our build
times were not that mindblowing either, saving roundabout ~100ms in wall
time. So sticking with the status quo felt like the better solution as
it is native to Meson.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 671df48 (meson: precompile "git-compat-util.h", 2026-03-19) we have introduced support for precompiled headers into Meson. At that time though we only converted "libgit.a" to make use of those. Nowadays though, our test-helper also consists of a bunch of code files, and all of these include "git-compat-util.h" via "test-tool.h" as the first header. So they're a natural target to also use precompiled headers. Adapt the test-tool executable to make use of them, which results in a surprisingly large speedup for clean builds: Benchmark 1: meson compile (version = HEAD~) Time (mean ± σ): 6.363 s ± 0.033 s [User: 92.858 s, System: 22.500 s] Range (min … max): 6.311 s … 6.418 s 10 runs Benchmark 2: meson compile (version = HEAD) Time (mean ± σ): 5.327 s ± 0.021 s [User: 75.135 s, System: 20.373 s] Range (min … max): 5.299 s … 5.362 s 10 runs Summary meson compile (version = HEAD) ran 1.19 ± 0.01 times faster than meson compile (version = HEAD~) Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Same as in the preceding commit, our unit tests don't use precompiled
headers yet. In this case though it's a tiny bit more complicated to
make use of them, as we do not want to include "git-compat-util.h" for
"clar.c", as that code file is a third-party implementation that is
independent of the Git codebase.
But there's an easy workaround: instead of linking that file into the
executable directly, we can easily adapt it to be built into a static
library first. Like that we can trivially have separate build flags for
that one file.
Do so and adapt the remaining sources to use precompiled headers. This
results in a small but noticeable build speedup:
Benchmark 1: meson compile (version = HEAD~)
Time (mean ± σ): 5.343 s ± 0.019 s [User: 75.478 s, System: 20.382 s]
Range (min … max): 5.308 s … 5.376 s 10 runs
Benchmark 2: meson compile (version = HEAD)
Time (mean ± σ): 5.077 s ± 0.017 s [User: 70.557 s, System: 19.999 s]
Range (min … max): 5.047 s … 5.103 s 10 runs
Summary
meson compile (version = HEAD) ran
1.05 ± 0.01 times faster than meson compile (version = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When using Meson 1.3.0 or newer, we use `fs.copyfile()` to put our completion helpers into the expected location so that our test suite can find these scripts. Naturally, we thus also add these scripts to our test dependencies so that we know to build them before executing tests. But there's an issue here: we include the "contrib/completion" subdir after we have already wired up our tests, so any dependencies we add here are not being honored correctly. This has the consequence that we don't know to copy around these completion helpers when we execute tests, and one has to manually `meson compile` beforehand. The interesting part here is that the code path we use with older versions of Meson don't suffer from the same problem as they use `configure_file()`, and that function will always run whenever the source file changes. It's conceptually correct to use `fs.copyfile()` instead, but given that it's mostly creating problems for us it does not really seem sensible to continue using it. Adapt the build instructions to unconditionally use `configure_file()` to fix this issue. A better fix would arguably be to promote our shell completion helpers out of "contrib/" -- they are an important part of Git nowadays, and these helpers get installed on lots of platforms. If so, we could also fix the order of subdir includes so that test dependencies are properly honored. But that feels like a bigger change, so that's left for a future patch series. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Our subproject wrappers are used on platforms that do not have the respective dependencies available. Most importantly, this can be used on Windows to have an almost-dependency-free build of Git. Update these wrappers via `meson wrap update`. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Starting with GitLab Runner 19.x, the runner executes `git credential reject` in its cleanup stage. This has bad interactions with our build environment because we install our own version of PortableGit, and the runner picks up that version of Git. The consequence is that we invoke PortableGit's default credential manager, which is Git Credential Manager for Windows. GCM then tries to use Windows Credential Manager, but it cannot and thus the job hangs in its cleanup phase forever. Fix this hang by unsetting the credential helper after installing PortableGit. This means that `git credential reject` becomes a no-op, and thus the cleanup succeeds again. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Hi, this patch series is a result from the discussions in [1]. It fixes a couple of smaller issues in our CI and bumps jobs that use EOL Docker images to instead use supported ones. Passing test runs can be found at [2] and [3] for GitLab and GitHub, respectively. Well, except for the Debian 12 HTTP/2 auth failures that Peff is already fixing in [4]. Note that I've also merged the Meson changes (ps/meson-improvements at ce4a600 (gitlab-ci: fix hanging MSVC jobs, 2026-09-24)) in there so that GitLab passes, but those are not strictly required as a dependency. Thanks! Patrick [1]: <20260906151137.GA328152@coredump.intra.peff.net> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/687 [3]: git#2445 [4]: <pull.2236.v2.git.1790283229626.gitgitgadget@gmail.com> To: git@vger.kernel.org Cc: Jeff King <peff@peff.net> Cc: Junio C Hamano <gitster@pobox.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 1, "change-id": "20261008-pks-ci-housekeeping-4cf7fac3b0df", "prefixes": [], "presubject": "" } }
|
There is a merge commit in this Pull Request: Please rebase the branch and force-push. |
Contributor
|
@pks-gitlab Great! |
One of the tests in t5004 extracts a ZIP file that contains some objects larger than 4GB and then double-checks whether we can read and archive such an object. That test has a bunch of prerequities: it requires a 64 bit `long`, unzip with 64-bit support and it only runs when EXPENSIVE is enabled. Consequently, not a lot of jobs even exercise this. One of the jobs that does run it though our Fedora-based job, as it ticks all the necessary boxes. But that job was silently broken: while the intent was to run on Fedora with breaking changes enabled, they are in fact disabled due to a typo. We're about to fix that typo in the next commit, but this will also uncover that the above test case is broken when running in SHA-256 repositories. The extracted objects are SHA-1 objects, so extracting them into a SHA-256 repository is not going to yield anything good. So once we fix the Fedora-based job to enable breaking changes, which will make tests use SHA-256 by default, the test will break. Fix this issue by adding the SHA1 prerequisite. Signed-off-by: Patrick Steinhardt <ps@pks.im>
The "fedora-breaking-changes-meson" job exercises Git with breaking changes enabled on Fedora with Meson. There's a typo in our CI scripts though, which has the effect that the build does not enable breaking changes by accident. Fix the typo. Signed-off-by: Patrick Steinhardt <ps@pks.im>
In d88d727 (ci: drop linux-clang job, 2023-06-01) we have dropped the "linux-clang" job because another job already uses Clang anyway. But we forgot to also drop the logic in "ci/run-build-and-tests.sh", so we now have some unused logic in there. Drop it. Signed-off-by: Patrick Steinhardt <ps@pks.im>
The linux32 job is used to exercise Git on a 32 bit platform. That job uses i386/ubuntu:20.04 though, and that version of Ubuntu is end of life nowadays. Furthermore, Ubuntu has dropped support for 32 bit entirely with the 20.04 release, so we cannot easily upgrade it to a more recent image anymore. Switch the job over to use i386/debian instead. Note that starting with Debian 13, support for i386 has been reduced [1]. But Debian still releases 32 bit Docker images for the latest release nowadays, so we will have coverage until at least 2030. [1]: https://www.debian.org/releases/trixie/release-notes/issues.en.html#i386-reduced-support Signed-off-by: Patrick Steinhardt <ps@pks.im>
The "linux-TEST-vars" job exercises Git with a bunch of non-default options enabled. The name of that job makes you want to cry though due to the weird upper-casing and because it doesn't really tell you what it even intends to do. Rename the job to "linux-exotic" instead. Signed-off-by: Patrick Steinhardt <ps@pks.im>
The "linux-exotic" job still uses Ubuntu 20.04, which is end of life nowadays and starting to show some cracks. Switch it over to use the "latest" tag instead so that we don't have to constantly update it anymore. Now arguably, this reduces test coverage for old versions of Ubuntu. But "latest" at least points to the most up-to-date LTS release of Ubuntu, compared to the "rolling" tag that uses the latest release regardless of the LTS status. So while "latest" and "rolling" are the same right now, that's not always the case. Furthermore, we have other jobs that test with ancient versions of Linux, like for example the one that uses AlmaLinux 8 (2021, originally tracking RHEL 8 from 2019) or Debian 12 (2023). Note that this also requires us to switch away from GCC 8, which is not supported by Ubuntu 26.04 anymore. Instead we simply use the default version of GCC. Signed-off-by: Patrick Steinhardt <ps@pks.im>
In the preceding commit we've dropped the last job that still used Ubuntu 20.04. We still had some conditional logic for that specific image that made us use Python 2 instead of Python 3, but this is dead code now. We could of course exercise Python 2 in any of our other CI jobs. But it reached end of life in 2020 already, and none of the distros that we use have it packaged anymore. Furthermore, it seems like the world has finally adapted to Python 3. So it doesn't feel all that useful to still exercise it. Drop the logic and instead use Python 3 unconditionally. Signed-off-by: Patrick Steinhardt <ps@pks.im>
The "linux-reftable" job exercises Git with reftables as its default backend. But this job is arguably redundant because we already have the "linux-reftable-leaks" job that exercises reftables with the leak sanitizer enabled, and it is unlikely that we will catch any extra bugs with the leak sanitizer disabled. Drop the job. Signed-off-by: Patrick Steinhardt <ps@pks.im>
pks-gitlab
force-pushed
the
b4/pks-ci-housekeeping
branch
from
October 8, 2026 08:00
e9f04d5 to
d903df5
Compare
|
There is a merge commit in this Pull Request: Please rebase the branch and force-push. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi,
this patch series is a result from the discussions in 1. It fixes a
couple of smaller issues in our CI and bumps jobs that use EOL Docker
images to instead use supported ones.
Test runs can be found at 2 and [3] for GitLab and GitHub,
respectively. Note that I've also merged the Meson changes
(ps/meson-improvements at ce4a600 (gitlab-ci: fix hanging MSVC jobs,
2026-09-24)) in there so that GitLab passes, but those are not strictly
required as a dependency.
Thanks!
Patrick
To: git@vger.kernel.org
Cc: Jeff King peff@peff.net
Cc: Junio C Hamano gitster@pobox.com
--- b4-submit-tracking ---
This section is used internally by b4 prep for tracking purposes.
{
"series": {
"revision": 1,
"change-id": "20261008-pks-ci-housekeeping-4cf7fac3b0df",
"prefixes": [],
"presubject": ""
}
}