Repository navigation
Enable PGO for Linux x86-64 ty releases - #4213
Conversation
## Summary This PR enables PGO for Ruff releases, starting with Linux x86-64. ### Design The release pipeline is modified as follows: - We build an instrumented, stripped release binary. - We run `check` and `format` on a corpus of projects from our ecosystem reports -- specifically, eight of the pinned ecosystem projects that we use in the ty CI. (In total, it's 502 Python and stub files.) - We merge the profiles via `llvm-profdata`. - We feed the result back into the existing `maturin` build. ### Results We evaluate performance on a held-out corpus: Prefect, Django, Pandas, scikit-learn, SciPy, and SymPy. Results are as follows: | Held-out project | `ruff check` | `ruff format` | | --- | ---: | ---: | | Django | 10.9% faster | 6.6% faster | | pandas | 16.3% faster | 9.7% faster | | scikit-learn | 10.8% faster | 7.9% faster | | SciPy | 14.7% faster | 5.6% faster | | SymPy | 17.1% faster | 11.4% faster | | Geometric mean | **14.0% faster** | **8.3% faster** | Beyond runtime: - **Binary size decreased by 6.2%** (27.96 MB to 26.23 MB). - **Release pipeline gets about 2x longer** (non-PGO release build took 7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s optimized wheel). ### Stack Additional platforms are covered in subsequent PRs in the stacked; platforms that are lower-priority at at-all difficult to run on natively are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS ARM, and Windows x86-64. I also attempted BOLT in #27588, but I've decided against pursuing that for now; see the results in that PR which speak for themselves. ty and uv will follow the same approach; see the stacks here: - astral-sh/ty#4213 - astral-sh/uv#21001 See: #7055.
|
How did you measure incremental edits? Our |
| # But it wasn't a highly optimized selection process. (For example, during | ||
| # development, we added Zulip and Warehouse, which reduced Ruff's CPU time by | ||
| # 0.35% while increasing its wheel size by 0.44%.) |
There was a problem hiding this comment.
Does this suggest we're overfitting? Can you run the benchmarks on homeassistant, airflow, prefect, DateType, static-frame, Freqtrade, and PyTorch?
The selection also seems to be the main difference to #3793. I used the entire ecosystem corpus and the results were disappointing. It looks like a bigger corpus is not a better corpus :)
There was a problem hiding this comment.
I don't think we're over-fitting. How would this suggest over-fitting? That's 0.35% vs. omitting those two projects, i.e., adding those two projects had only a moderately positive effect.
There was a problem hiding this comment.
I read that wrong. I read it as it increased walltime. I'd still be interested in some more benchmarks, because I remember some of them regressed when I did PGO (I'm sure yours is much better).
I think homeassistant is worth it, and some of the projects at the very bottom (or you can go all in and run it over all ecosystem projects)
ty PGO manual walltime report
Historical detailed benchmark table recovered from the archived report. This predates the later PGO training-timeout correction, so use the later matrix for the final conclusion.
- Configuration: current default ty parallelism
- Repetitions: 3 warmups and 10 measured runs per binary
- Aggregate: release 76.683s; PGO release 73.128s; delta -4.64%; speedup 1.049× across 162 projects
- homeassistant: release 4.015s; PGO release 4.049s; delta +0.85%; speedup 0.992×
| Project | Release | PGO release | Delta | Speedup |
|---|---|---|---|---|
| pandera | 1.308s | 1.116s | -14.70% | 1.172x |
| manticore | 2.537s | 2.181s | -14.04% | 1.163x |
| spark | 1.240s | 1.074s | -13.43% | 1.155x |
| setuptools | 0.487s | 0.429s | -11.95% | 1.136x |
| schema_salad | 0.303s | 0.268s | -11.55% | 1.131x |
| mitmproxy | 0.331s | 0.295s | -10.87% | 1.122x |
| pip | 0.549s | 0.490s | -10.73% | 1.120x |
| altair | 0.338s | 0.303s | -10.53% | 1.118x |
| numpy-stl | 0.289s | 0.259s | -10.29% | 1.115x |
| egglog-python | 0.317s | 0.284s | -10.29% | 1.115x |
| pydantic | 0.801s | 0.721s | -9.93% | 1.110x |
| nionutils | 0.195s | 0.176s | -9.70% | 1.107x |
| aioredis | 0.240s | 0.217s | -9.42% | 1.104x |
| colour | 9.807s | 8.887s | -9.38% | 1.104x |
| isort | 0.584s | 0.530s | -9.25% | 1.102x |
| python-chess | 0.228s | 0.207s | -9.06% | 1.100x |
| black | 0.170s | 0.155s | -8.81% | 1.097x |
| AutoSplit | 0.212s | 0.194s | -8.75% | 1.096x |
| hydra-zen | 0.227s | 0.207s | -8.73% | 1.096x |
| cryptography | 0.384s | 0.350s | -8.70% | 1.095x |
| static-frame | 0.886s | 0.809s | -8.67% | 1.095x |
| pyp | 0.128s | 0.117s | -8.39% | 1.092x |
| jinja | 0.171s | 0.157s | -8.29% | 1.090x |
| koda-validate | 0.129s | 0.118s | -8.28% | 1.090x |
| aiortc | 0.144s | 0.133s | -8.23% | 1.090x |
| flake8-pyi | 0.143s | 0.131s | -8.17% | 1.089x |
| mkosi | 0.257s | 0.236s | -8.10% | 1.088x |
| pytest-robotframework | 0.238s | 0.219s | -8.07% | 1.088x |
| pandas-stubs | 2.941s | 2.706s | -8.00% | 1.087x |
| zope.interface | 0.145s | 0.134s | -7.97% | 1.087x |
| docstring-adder | 0.168s | 0.155s | -7.96% | 1.086x |
| pwndbg | 0.436s | 0.401s | -7.93% | 1.086x |
| anyio | 0.190s | 0.175s | -7.86% | 1.085x |
| operator | 0.203s | 0.187s | -7.83% | 1.085x |
| attrs | 0.176s | 0.162s | -7.68% | 1.083x |
| more-itertools | 0.238s | 0.220s | -7.55% | 1.082x |
| Tanjun | 0.204s | 0.189s | -7.52% | 1.081x |
| build | 0.120s | 0.111s | -7.46% | 1.081x |
| pyodide | 0.315s | 0.292s | -7.41% | 1.080x |
| imagehash | 0.204s | 0.189s | -7.34% | 1.079x |
| mkdocs | 0.169s | 0.156s | -7.20% | 1.078x |
| dedupe | 0.156s | 0.145s | -7.18% | 1.077x |
| ecosystem-analyzer | 0.155s | 0.144s | -7.12% | 1.077x |
| pylint | 0.196s | 0.182s | -6.96% | 1.075x |
| antidote | 0.202s | 0.188s | -6.92% | 1.074x |
| steam.py | 0.316s | 0.295s | -6.87% | 1.074x |
| stone | 0.174s | 0.162s | -6.80% | 1.073x |
| vision | 0.467s | 0.436s | -6.75% | 1.072x |
| xarray | 0.852s | 0.796s | -6.61% | 1.071x |
| parso | 0.187s | 0.175s | -6.54% | 1.070x |
| arviz | 0.284s | 0.266s | -6.36% | 1.068x |
| Expression | 0.154s | 0.144s | -6.23% | 1.066x |
| pyproject-metadata | 0.120s | 0.112s | -6.22% | 1.066x |
| pytest-autoprofile | 0.139s | 0.131s | -6.16% | 1.066x |
| strawberry | 0.171s | 0.160s | -6.13% | 1.065x |
| mypy_primer | 0.123s | 0.115s | -6.08% | 1.065x |
| jax | 0.984s | 0.925s | -6.07% | 1.065x |
| pycryptodome | 0.300s | 0.282s | -5.96% | 1.063x |
| typeshed-stats | 0.196s | 0.184s | -5.88% | 1.062x |
| mypy-protobuf | 0.171s | 0.161s | -5.64% | 1.060x |
| xarray-dataclasses | 0.382s | 0.360s | -5.62% | 1.060x |
| kopf | 0.135s | 0.128s | -5.60% | 1.059x |
| freqtrade | 0.497s | 0.469s | -5.58% | 1.059x |
| trio | 0.255s | 0.241s | -5.57% | 1.059x |
| svcs | 0.100s | 0.095s | -5.54% | 1.059x |
| async-utils | 0.120s | 0.113s | -5.52% | 1.058x |
| paroxython | 0.101s | 0.096s | -5.46% | 1.058x |
| PyGithub | 0.249s | 0.235s | -5.33% | 1.056x |
| comtypes | 0.180s | 0.170s | -5.21% | 1.055x |
| streamlit | 0.260s | 0.247s | -5.09% | 1.054x |
| itsdangerous | 0.078s | 0.074s | -5.07% | 1.053x |
| schemathesis | 0.332s | 0.315s | -5.06% | 1.053x |
| DateType | 0.102s | 0.097s | -5.03% | 1.053x |
| graphql-core | 0.313s | 0.298s | -4.98% | 1.052x |
| pyinstrument | 0.102s | 0.097s | -4.95% | 1.052x |
| rclip | 0.127s | 0.121s | -4.95% | 1.052x |
| alerta | 0.175s | 0.166s | -4.89% | 1.051x |
| pylox | 0.116s | 0.111s | -4.86% | 1.051x |
| pyjwt | 0.109s | 0.104s | -4.76% | 1.050x |
| yarl | 0.116s | 0.111s | -4.73% | 1.050x |
| pyppeteer | 0.122s | 0.116s | -4.69% | 1.049x |
| pegen | 0.090s | 0.086s | -4.66% | 1.049x |
| beartype | 0.197s | 0.188s | -4.65% | 1.049x |
| poetry | 0.250s | 0.239s | -4.59% | 1.048x |
| PyWinCtl | 0.116s | 0.111s | -4.57% | 1.048x |
| bidict | 0.091s | 0.087s | -4.56% | 1.048x |
| archinstall | 0.262s | 0.250s | -4.55% | 1.048x |
| bandersnatch | 0.131s | 0.125s | -4.55% | 1.048x |
| httpx-caching | 0.112s | 0.107s | -4.36% | 1.046x |
| janus | 0.113s | 0.108s | -4.31% | 1.045x |
| nox | 0.108s | 0.103s | -4.20% | 1.044x |
| zipp | 0.089s | 0.085s | -4.18% | 1.044x |
| prefect | 1.709s | 1.639s | -4.09% | 1.043x |
| sphinx | 0.383s | 0.367s | -3.98% | 1.041x |
| urllib3 | 0.186s | 0.179s | -3.97% | 1.041x |
| python-sop | 0.110s | 0.106s | -3.94% | 1.041x |
| cloud-init | 0.505s | 0.485s | -3.92% | 1.041x |
| git-revise | 0.106s | 0.102s | -3.88% | 1.040x |
| speedrun.com_global_scoreboard_webapp | 0.120s | 0.116s | -3.87% | 1.040x |
| werkzeug | 0.183s | 0.176s | -3.84% | 1.040x |
| alectryon | 0.085s | 0.082s | -3.82% | 1.040x |
| starlette | 0.131s | 0.126s | -3.81% | 1.040x |
| boostedblob | 0.140s | 0.134s | -3.80% | 1.040x |
| paasta | 0.250s | 0.240s | -3.79% | 1.039x |
| dulwich | 0.359s | 0.346s | -3.73% | 1.039x |
| cwltool | 0.230s | 0.221s | -3.73% | 1.039x |
| meson | 0.599s | 0.577s | -3.66% | 1.038x |
| artigraph | 0.164s | 0.158s | -3.64% | 1.038x |
| cibuildwheel | 0.116s | 0.112s | -3.62% | 1.038x |
| discord.py | 0.314s | 0.303s | -3.55% | 1.037x |
| sockeye | 0.895s | 0.864s | -3.52% | 1.036x |
| CPython (cases_generator) | 0.900s | 0.869s | -3.40% | 1.035x |
| dacite | 0.075s | 0.072s | -3.37% | 1.035x |
| CPython (peg_generator) | 0.103s | 0.100s | -3.35% | 1.035x |
| django-modern-rest | 0.161s | 0.155s | -3.35% | 1.035x |
| twine | 0.115s | 0.111s | -3.32% | 1.034x |
| flake8 | 0.107s | 0.103s | -3.31% | 1.034x |
| aiohttp | 0.265s | 0.257s | -3.26% | 1.034x |
| packaging | 0.108s | 0.105s | -3.03% | 1.031x |
| ppb-vector | 0.087s | 0.084s | -2.96% | 1.031x |
| materialize | 0.436s | 0.423s | -2.95% | 1.030x |
| check-jsonschema | 0.099s | 0.096s | -2.93% | 1.030x |
| cki-lib | 0.122s | 0.119s | -2.89% | 1.030x |
| mypy | 1.049s | 1.019s | -2.87% | 1.030x |
| porcupine | 0.123s | 0.120s | -2.84% | 1.029x |
| websockets | 0.122s | 0.119s | -2.81% | 1.029x |
| hydpy | 0.598s | 0.582s | -2.73% | 1.028x |
| asynq | 0.105s | 0.102s | -2.61% | 1.027x |
| aiohttp-devtools | 0.892s | 0.869s | -2.60% | 1.027x |
| com2ann | 0.893s | 0.870s | -2.58% | 1.027x |
| bokeh | 0.313s | 0.306s | -2.48% | 1.025x |
| downforeveryone | 0.087s | 0.084s | -2.48% | 1.025x |
| zulip | 0.953s | 0.930s | -2.47% | 1.025x |
| ibis | 0.762s | 0.744s | -2.38% | 1.024x |
| rich | 0.140s | 0.137s | -2.34% | 1.024x |
| dragonchain | 0.128s | 0.126s | -2.12% | 1.022x |
| tornado | 0.208s | 0.204s | -2.06% | 1.021x |
| psycopg | 0.246s | 0.241s | -1.99% | 1.020x |
| pybind11 | 0.122s | 0.120s | -1.87% | 1.019x |
| python-htmlgen | 0.084s | 0.082s | -1.77% | 1.018x |
| sympy | 3.226s | 3.174s | -1.62% | 1.016x |
| dd-trace-py | 1.450s | 1.426s | -1.61% | 1.016x |
| scrapy | 0.267s | 0.263s | -1.37% | 1.014x |
| rotki | 0.799s | 0.789s | -1.28% | 1.013x |
| ignite | 0.284s | 0.281s | -1.20% | 1.012x |
| scipy-stubs | 0.535s | 0.529s | -1.14% | 1.012x |
| CPython (Argument Clinic) | 0.880s | 0.872s | -0.94% | 1.009x |
| pywin32 | 0.493s | 0.490s | -0.70% | 1.007x |
| django-test-migrations | 0.090s | 0.090s | -0.68% | 1.007x |
| spack | 0.709s | 0.705s | -0.65% | 1.007x |
| django-stubs | 0.230s | 0.229s | -0.61% | 1.006x |
| scikit-build-core | 0.140s | 0.140s | -0.45% | 1.005x |
| pytest | 0.258s | 0.257s | -0.41% | 1.004x |
| openlibrary | 0.273s | 0.272s | -0.30% | 1.003x |
| kornia | 0.250s | 0.251s | +0.59% | 0.994x |
| core | 3.281s | 3.302s | +0.64% | 0.994x |
| mongo-python-driver | 0.272s | 0.275s | +1.01% | 0.990x |
| pandas | 2.317s | 2.345s | +1.22% | 0.988x |
| optuna | 0.396s | 0.415s | +4.64% | 0.956x |
| apprise | 0.435s | 0.457s | +5.02% | 0.952x |
| scikit-learn | 1.357s | 1.448s | +6.74% | 0.937x |
| scipy | 3.722s | 4.044s | +8.64% | 0.920x |
MichaReiser
left a comment
There was a problem hiding this comment.
Thank you. I suggest running the benchmarks on a few more projects to ensure we're not overfitting a certain set of projects
| "--project", | ||
| str(checkout), | ||
| "--python-version", | ||
| PYTHON_VERSION, |
There was a problem hiding this comment.
What's the reason for hardcoding the python version. Isn't this penalizing projects using older Python versions? Did you run your benchmarks with the project's real Python version?
There was a problem hiding this comment.
For consistency, but I can change it.
|
No, they're omitted from training and evaluation. It's just much simpler that way. |
|
I'll redo the analysis both ways. |
|
We do have a I think, at least, we need to do the evaluation with dependencies installed. |
|
Yup, makes sense. |
|
Good news all around... The current pipeline doesn't regress CPU time in any of the ecosystem projects. All 162 projects showed improved CPU time (20.7% lower on average), two projects showed wall-time regressions that could be noise (18.8% faster on average). All the projects that regressed in your report now improve by 10-20%. Adding Home Assistant, or Home Assistant and SciPy, had almost no effect: On the dependency front, even the existing pipeline shows an enormous improvement when evaluating over "projects with dependencies installed", but it gets even better if you train with dependencies installed too: |
|
Sorry. I didn't mean to add homeassistant to the training corpus. Only adding it to the evolution corpus |
|
Oh np. With current PGO it does improve by 18% or so IIRC from the session. I also modified the script to use the project Python version. |
| EcosystemProject( | ||
| name="astropy", | ||
| repository="astropy/astropy", | ||
| revision="b779108c7cec25c840c0f744fdf2a1550441e309", | ||
| source_directories=("astropy/units",), | ||
| python_version="3.11", | ||
| ), |
There was a problem hiding this comment.
Any reason why you aren't using setup_primer_project? It also has an option to print the ty invocation command.
| "--no-progress", | ||
| "-qq", |
There was a problem hiding this comment.
Not important, but I believe -q implies --no-progress
| try: | ||
| initialized = request( |
There was a problem hiding this comment.
I'm waiting for the day when we'll break this 😆
I, for sure, will not think of our PGO pipeline when considering breaking changes
There was a problem hiding this comment.
I need to setup CI for this...
There was a problem hiding this comment.
It might be simpler in ty, because we only update ty in release builds (can it run at the same time as when we run our other release jobs?)
88658d8 to
58f75f6
Compare
## Summary This PR enables PGO for ty's macOS ARM64 releases, following the approach outlined in #4213. The existing linker configuration and safe identical-code folding are preserved. Across seven held-out macOS projects, `ty check` finishes 5.0% faster and uses 7.1% less CPU; incremental language-server edits are 7.3% faster. The production executable is 3.6% smaller than without PGO, although the wheel and release archive are each 2.8% larger. The macOS ARM64 release job uses the same Namespace runner as Ruff, reducing its full-stack PGO build from 23m 47s to 6m 39s (72%). Three Namespace builds finished in 6m 38s to 7m 59s; the previous Depot runner took 11m 20s.
## Summary This PR enables PGO for ty's Windows x86-64 releases, following the approach outlined in #4213. Both builds preserve static Visual C++ runtime linking. On the Windows holdout set, `ty check` uses 19.7% less CPU and finishes 17.5% faster, while incremental language-server edits get 14.7% faster. The release binary gets 9.6% smaller, and the wheel and release archive each get 3.4% smaller. Only the Windows x86-64 PGO job uses the same 16-core Namespace runner as Ruff, reducing the full-stack build from 26m 08s to 12m 24s (53%). The previous 32-core Depot runner took 14m 17s. With Linux x86-64 at 11m 28s and Linux ARM64 at 11m 37s, Windows is now the release bottleneck. The remaining Windows targets keep their existing runners.
## Summary This PR enables PGO for ty's Linux ARM64 releases, following the approach outlined in #4213. (To enable PGO, we also move to a native ARM64 runner.) The existing jemalloc page-size configuration is preserved. Across four held-out Linux ARM64 projects, `ty check` uses 15.0% less CPU and finishes 11.9% faster. The release binary gets 4.9% smaller, the wheel gets 1.2% smaller, and the release archive gets 1.4% smaller. The Linux ARM64 PGO build uses an eight-core Depot runner, reducing its runtime from 14m 51s on a four-core GitHub runner to 11m 37s (22%). In the matched full-stack run, eight-core Linux x86-64 finished in 11m 28s and Namespace Windows finished in 12m 24s, making Windows the release bottleneck.
## Summary This PR enables PGO for Ruff releases, starting with Linux x86-64. ### Design The release pipeline is modified as follows: - We build an instrumented, stripped release binary. - We run `check` and `format` on a corpus of projects from our ecosystem reports -- specifically, eight of the pinned ecosystem projects that we use in the ty CI. (In total, it's 502 Python and stub files.) - We merge the profiles via `llvm-profdata`. - We feed the result back into the existing `maturin` build. ### Results We evaluate performance on a held-out corpus: Prefect, Django, Pandas, scikit-learn, SciPy, and SymPy. Results are as follows: | Held-out project | `ruff check` | `ruff format` | | --- | ---: | ---: | | Django | 10.9% faster | 6.6% faster | | pandas | 16.3% faster | 9.7% faster | | scikit-learn | 10.8% faster | 7.9% faster | | SciPy | 14.7% faster | 5.6% faster | | SymPy | 17.1% faster | 11.4% faster | | Geometric mean | **14.0% faster** | **8.3% faster** | Beyond runtime: - **Binary size decreased by 6.2%** (27.96 MB to 26.23 MB). - **Release pipeline gets about 2x longer** (non-PGO release build took 7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s optimized wheel). ### Stack Additional platforms are covered in subsequent PRs in the stacked; platforms that are lower-priority at at-all difficult to run on natively are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS ARM, and Windows x86-64. I also attempted BOLT in astral-sh#27588, but I've decided against pursuing that for now; see the results in that PR which speak for themselves. ty and uv will follow the same approach; see the stacks here: - astral-sh/ty#4213 - astral-sh/uv#21001 See: astral-sh#7055.
## Summary This PR enables PGO for uv releases, starting with Linux x86-64. ### Design The release pipeline is modified as follows: - We build instrumented `uv` and `uvx` release binaries inside the existing manylinux container. - We train directly on eleven shared `test/ecosystem` fixtures: cibuildwheel, Cookiecutter, Flask, HTTPX, LLM, the OpenAI Python SDK, Poetry, pytest-cov, Sentry, Zulip, and the 38-project pyx workspace. - Every project exercises cold and warm `uv pip compile` resolution, cold and warm `uv lock` resolution, `uv export`, `uv pip` installation, and `uv sync` installation on every release platform. - Per-project `exclude-dependencies` settings omit packages without compatible release-platform wheels while preserving the rest of each real dependency graph. - Raw profiles are merged online by workload family, then merged with `llvm-profdata` and fed back into the existing `maturin` build. - JupyterLab, Saleor, Semantic Kernel, Transformers, and Warehouse remain separate, held-out evaluation projects. ### Results We benchmarked the builds on Linux x86-64 using the expanded corpus for training and five separate held-out ecosystem projects for evaluation. | Held-out project | Dependency resolution | Wheel installation | Project locking | Lockfile export | | ---------------- | --------------------: | -----------------: | ---------------: | --------------: | | JupyterLab | 11.2% faster | 0.0% | 10.1% faster | 10.6% faster | | Semantic Kernel | 9.3% faster | 7.1% faster | 12.5% faster | 8.7% faster | | Transformers | 11.7% faster | 4.1% faster | 9.5% faster | 11.0% faster | | Saleor | — | — | 9.7% faster | 10.0% faster | | Warehouse | — | — | 9.7% faster | 7.6% faster | | Geometric mean | **10.7% faster** | **3.7% faster** | **10.3% faster** | **9.6% faster** | Across all sixteen held-out workloads, wall time decreased by **8.9%** (95% CI: 8.4–9.8%) and CPU time decreased by **8.4%** (95% CI: 7.9–9.0%). The matched Linux executable was **13.3% smaller** (58.91 MB to 51.05 MB), or **12.3% smaller** after gzip compression (22.68 MB to 19.90 MB). (Saleor and Warehouse include source-only dependencies, so they are benchmarked on locking and export rather than wheel-only installation.) ### Stack Additional platforms are covered in subsequent PRs in the stack: macOS ARM64, Windows x86-64, and Linux ARM64. Ruff and ty follow the same approach; see the stacks here: - astral-sh/ruff#27570 - astral-sh/ty#4213
This PR enables PGO for uv releases, starting with Linux x86-64. The release pipeline is modified as follows: - We build instrumented `uv` and `uvx` release binaries inside the existing manylinux container. - We train directly on eleven shared `test/ecosystem` fixtures: cibuildwheel, Cookiecutter, Flask, HTTPX, LLM, the OpenAI Python SDK, Poetry, pytest-cov, Sentry, Zulip, and the 38-project pyx workspace. - Every project exercises cold and warm `uv pip compile` resolution, cold and warm `uv lock` resolution, `uv export`, `uv pip` installation, and `uv sync` installation on every release platform. - Per-project `exclude-dependencies` settings omit packages without compatible release-platform wheels while preserving the rest of each real dependency graph. - Raw profiles are merged online by workload family, then merged with `llvm-profdata` and fed back into the existing `maturin` build. - JupyterLab, Saleor, Semantic Kernel, Transformers, and Warehouse remain separate, held-out evaluation projects. We benchmarked the builds on Linux x86-64 using the expanded corpus for training and five separate held-out ecosystem projects for evaluation. | Held-out project | Dependency resolution | Wheel installation | Project locking | Lockfile export | | ---------------- | --------------------: | -----------------: | ---------------: | --------------: | | JupyterLab | 11.2% faster | 0.0% | 10.1% faster | 10.6% faster | | Semantic Kernel | 9.3% faster | 7.1% faster | 12.5% faster | 8.7% faster | | Transformers | 11.7% faster | 4.1% faster | 9.5% faster | 11.0% faster | | Saleor | — | — | 9.7% faster | 10.0% faster | | Warehouse | — | — | 9.7% faster | 7.6% faster | | Geometric mean | **10.7% faster** | **3.7% faster** | **10.3% faster** | **9.6% faster** | Across all sixteen held-out workloads, wall time decreased by **8.9%** (95% CI: 8.4–9.8%) and CPU time decreased by **8.4%** (95% CI: 7.9–9.0%). The matched Linux executable was **13.3% smaller** (58.91 MB to 51.05 MB), or **12.3% smaller** after gzip compression (22.68 MB to 19.90 MB). (Saleor and Warehouse include source-only dependencies, so they are benchmarked on locking and export rather than wheel-only installation.) Additional platforms are covered in subsequent PRs in the stack: macOS ARM64, Windows x86-64, and Linux ARM64. Ruff and ty follow the same approach; see the stacks here: - astral-sh/ruff#27570 - astral-sh/ty#4213
Summary
This PR enables PGO for ty releases, starting with Linux x86-64. The approach follows that outlined in astral-sh/ruff#27570.
Design
The release pipeline is modified as follows:
ty checkon the same ten pinned ecosystem projects used by Ruff. In total, the corpus contains 618 Python and stub files.ty serveragainst a temporary project, covering diagnostics, hover, go-to-definition, completion, and incremental cross-file edits.llvm-profdata.maturinbuild.Results
Here's the initial, untuned PGO performance on seven held-out projects (and language-server performance on three held-out projects):
ty checkAs an aside, profiling the language server explicitly (i.e., including the language server commands in the PGO training) improved incremental-edit latency by another 7.6% over CLI-only PGO; without that training, incremental edit performance was still up (but not quite as much).
Beyond raw performance, with the tuned PGO settings, we see the following results:
Stack
Additional platforms are covered in subsequent PRs:
Ruff and uv follow the same approach; see astral-sh/ruff#27570 and astral-sh/uv#21001.