Skip to content

Enable PGO for Linux x86-64 ty releases - #4213

Merged
charliermarsh merged 11 commits into
mainfrom
charlie/ty-linux-pgo-prototype
Aug 11, 2026
Merged

charliermarsh merged 11 commits into
mainfrom
charlie/ty-linux-pgo-prototype

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Aug 7, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR enables PGO for ty releases, starting with Linux x86-64. The approach follows that outlined in astral-sh/ruff#27570.

Design

The release pipeline is modified as follows:

  • We build an instrumented release binary using the Rust toolchain pinned by the Ruff submodule.
  • We run ty check on the same ten pinned ecosystem projects used by Ruff. In total, the corpus contains 618 Python and stub files.
  • We also exercise ty server against a temporary project, covering diagnostics, hover, go-to-definition, completion, and incremental cross-file edits.
  • We merge the profiles via llvm-profdata.
  • We derive LLVM's hot-code threshold from the profile's 95th percentile, matching its size-optimization threshold and avoiding unnecessary expansion of moderately hot functions. (Without this, performance was marginally better, but wheel size increased by 5-10%!)
  • We feed the result back into the existing maturin build.

Results

Here's the initial, untuned PGO performance on seven held-out projects (and language-server performance on three held-out projects):

Held-out project ty check Incremental edits
Black 7.1% faster 17.1% faster
isort 13.3% faster 18.1% faster
Jinja 5.5% faster 13.7% faster
Django 25.5% faster —
pandas 15.9% faster —
scikit-learn 19.6% faster —
SymPy 21.5% faster —
Geometric mean 15.8% faster 16.3% faster

As an aside, profiling the language server explicitly (i.e., including the language server commands in the PGO training) improved incremental-edit latency by another 7.6% over CLI-only PGO; without that training, incremental edit performance was still up (but not quite as much).

Beyond raw performance, with the tuned PGO settings, we see the following results:

  • Binary size decreased by 7.98% (28.10 MB to 25.86 MB compared with non-PGO).
  • Wheel size decreased by 1.32% (12.71 MB to 12.54 MB compared with non-PGO), and by 14.11% compared with untuned PGO.
  • Release archive size decreased by 1.53% (12.35 MB to 12.16 MB compared with non-PGO), and by 14.25% compared with untuned PGO.
  • Incremental language-server edits are 3.30% faster than untuned PGO.
  • Linux x86-64 release build finishes in 11m28s on eight-core Depot, compared with 12m59s on four-core Depot and 15m45s on the previous GitHub runner. The full-stack bottleneck shifts to Windows at 12m24s; other Linux targets retain their four-core runners.

Stack

Additional platforms are covered in subsequent PRs:

Ruff and uv follow the same approach; see astral-sh/ruff#27570 and astral-sh/uv#21001.

@charliermarsh charliermarsh changed the title Enable profile-guided optimization for Linux x86_64 builds Enable PGO for Linux x86-64 ty releases Aug 7, 2026
charliermarsh added a commit to astral-sh/ruff that referenced this pull request Aug 10, 2026
## Summary

This PR enables PGO for Ruff releases, starting with Linux x86-64.

### Design

The release pipeline is modified as follows:

- We build an instrumented, stripped release binary.
- We run `check` and `format` on a corpus of projects from our ecosystem
reports -- specifically, eight of the pinned ecosystem projects that we
use in the ty CI. (In total, it's 502 Python and stub files.)
- We merge the profiles via `llvm-profdata`.
- We feed the result back into the existing `maturin` build.

### Results

We evaluate performance on a held-out corpus: Prefect, Django, Pandas,
scikit-learn, SciPy, and SymPy.

Results are as follows:

| Held-out project | `ruff check` | `ruff format` |
| --- | ---: | ---: |
| Django | 10.9% faster | 6.6% faster |
| pandas | 16.3% faster | 9.7% faster |
| scikit-learn | 10.8% faster | 7.9% faster |
| SciPy | 14.7% faster | 5.6% faster |
| SymPy | 17.1% faster | 11.4% faster |
| Geometric mean | **14.0% faster** | **8.3% faster** |

Beyond runtime:

- **Binary size decreased by 6.2%** (27.96 MB to 26.23 MB).
- **Release pipeline gets about 2x longer** (non-PGO release build took
7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s
optimized wheel).

### Stack

Additional platforms are covered in subsequent PRs in the stacked;
platforms that are lower-priority at at-all difficult to run on natively
are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS
ARM, and Windows x86-64.

I also attempted BOLT in #27588,
but I've decided against pursuing that for now; see the results in that
PR which speak for themselves.

ty and uv will follow the same approach; see the stacks here:

- astral-sh/ty#4213
- astral-sh/uv#21001

See: #7055.
@charliermarsh charliermarsh added the release Related to the release process label Aug 10, 2026
@charliermarsh
charliermarsh marked this pull request as ready for review August 10, 2026 23:16
@MichaReiser

Copy link
Copy Markdown
Member

How did you measure incremental edits? Our file_check[hot] benchmark is a fairly made up benchmark, it doesn't measure a realistic or even meaningful file change.

Comment thread scripts/build_ty_pgo.py
Comment on lines +65 to +67
# But it wasn't a highly optimized selection process. (For example, during
# development, we added Zulip and Warehouse, which reduced Ruff's CPU time by
# 0.35% while increasing its wheel size by 0.44%.)

@MichaReiser MichaReiser Aug 11, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this suggest we're overfitting? Can you run the benchmarks on homeassistant, airflow, prefect, DateType, static-frame, Freqtrade, and PyTorch?

The selection also seems to be the main difference to #3793. I used the entire ecosystem corpus and the results were disappointing. It looks like a bigger corpus is not a better corpus :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we're over-fitting. How would this suggest over-fitting? That's 0.35% vs. omitting those two projects, i.e., adding those two projects had only a moderately positive effect.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read that wrong. I read it as it increased walltime. I'd still be interested in some more benchmarks, because I remember some of them regressed when I did PGO (I'm sure yours is much better).

I think homeassistant is worth it, and some of the projects at the very bottom (or you can go all in and run it over all ecosystem projects)


ty PGO manual walltime report

Historical detailed benchmark table recovered from the archived report. This predates the later PGO training-timeout correction, so use the later matrix for the final conclusion.

  • Configuration: current default ty parallelism
  • Repetitions: 3 warmups and 10 measured runs per binary
  • Aggregate: release 76.683s; PGO release 73.128s; delta -4.64%; speedup 1.049× across 162 projects
  • homeassistant: release 4.015s; PGO release 4.049s; delta +0.85%; speedup 0.992×
Project Release PGO release Delta Speedup
pandera 1.308s 1.116s -14.70% 1.172x
manticore 2.537s 2.181s -14.04% 1.163x
spark 1.240s 1.074s -13.43% 1.155x
setuptools 0.487s 0.429s -11.95% 1.136x
schema_salad 0.303s 0.268s -11.55% 1.131x
mitmproxy 0.331s 0.295s -10.87% 1.122x
pip 0.549s 0.490s -10.73% 1.120x
altair 0.338s 0.303s -10.53% 1.118x
numpy-stl 0.289s 0.259s -10.29% 1.115x
egglog-python 0.317s 0.284s -10.29% 1.115x
pydantic 0.801s 0.721s -9.93% 1.110x
nionutils 0.195s 0.176s -9.70% 1.107x
aioredis 0.240s 0.217s -9.42% 1.104x
colour 9.807s 8.887s -9.38% 1.104x
isort 0.584s 0.530s -9.25% 1.102x
python-chess 0.228s 0.207s -9.06% 1.100x
black 0.170s 0.155s -8.81% 1.097x
AutoSplit 0.212s 0.194s -8.75% 1.096x
hydra-zen 0.227s 0.207s -8.73% 1.096x
cryptography 0.384s 0.350s -8.70% 1.095x
static-frame 0.886s 0.809s -8.67% 1.095x
pyp 0.128s 0.117s -8.39% 1.092x
jinja 0.171s 0.157s -8.29% 1.090x
koda-validate 0.129s 0.118s -8.28% 1.090x
aiortc 0.144s 0.133s -8.23% 1.090x
flake8-pyi 0.143s 0.131s -8.17% 1.089x
mkosi 0.257s 0.236s -8.10% 1.088x
pytest-robotframework 0.238s 0.219s -8.07% 1.088x
pandas-stubs 2.941s 2.706s -8.00% 1.087x
zope.interface 0.145s 0.134s -7.97% 1.087x
docstring-adder 0.168s 0.155s -7.96% 1.086x
pwndbg 0.436s 0.401s -7.93% 1.086x
anyio 0.190s 0.175s -7.86% 1.085x
operator 0.203s 0.187s -7.83% 1.085x
attrs 0.176s 0.162s -7.68% 1.083x
more-itertools 0.238s 0.220s -7.55% 1.082x
Tanjun 0.204s 0.189s -7.52% 1.081x
build 0.120s 0.111s -7.46% 1.081x
pyodide 0.315s 0.292s -7.41% 1.080x
imagehash 0.204s 0.189s -7.34% 1.079x
mkdocs 0.169s 0.156s -7.20% 1.078x
dedupe 0.156s 0.145s -7.18% 1.077x
ecosystem-analyzer 0.155s 0.144s -7.12% 1.077x
pylint 0.196s 0.182s -6.96% 1.075x
antidote 0.202s 0.188s -6.92% 1.074x
steam.py 0.316s 0.295s -6.87% 1.074x
stone 0.174s 0.162s -6.80% 1.073x
vision 0.467s 0.436s -6.75% 1.072x
xarray 0.852s 0.796s -6.61% 1.071x
parso 0.187s 0.175s -6.54% 1.070x
arviz 0.284s 0.266s -6.36% 1.068x
Expression 0.154s 0.144s -6.23% 1.066x
pyproject-metadata 0.120s 0.112s -6.22% 1.066x
pytest-autoprofile 0.139s 0.131s -6.16% 1.066x
strawberry 0.171s 0.160s -6.13% 1.065x
mypy_primer 0.123s 0.115s -6.08% 1.065x
jax 0.984s 0.925s -6.07% 1.065x
pycryptodome 0.300s 0.282s -5.96% 1.063x
typeshed-stats 0.196s 0.184s -5.88% 1.062x
mypy-protobuf 0.171s 0.161s -5.64% 1.060x
xarray-dataclasses 0.382s 0.360s -5.62% 1.060x
kopf 0.135s 0.128s -5.60% 1.059x
freqtrade 0.497s 0.469s -5.58% 1.059x
trio 0.255s 0.241s -5.57% 1.059x
svcs 0.100s 0.095s -5.54% 1.059x
async-utils 0.120s 0.113s -5.52% 1.058x
paroxython 0.101s 0.096s -5.46% 1.058x
PyGithub 0.249s 0.235s -5.33% 1.056x
comtypes 0.180s 0.170s -5.21% 1.055x
streamlit 0.260s 0.247s -5.09% 1.054x
itsdangerous 0.078s 0.074s -5.07% 1.053x
schemathesis 0.332s 0.315s -5.06% 1.053x
DateType 0.102s 0.097s -5.03% 1.053x
graphql-core 0.313s 0.298s -4.98% 1.052x
pyinstrument 0.102s 0.097s -4.95% 1.052x
rclip 0.127s 0.121s -4.95% 1.052x
alerta 0.175s 0.166s -4.89% 1.051x
pylox 0.116s 0.111s -4.86% 1.051x
pyjwt 0.109s 0.104s -4.76% 1.050x
yarl 0.116s 0.111s -4.73% 1.050x
pyppeteer 0.122s 0.116s -4.69% 1.049x
pegen 0.090s 0.086s -4.66% 1.049x
beartype 0.197s 0.188s -4.65% 1.049x
poetry 0.250s 0.239s -4.59% 1.048x
PyWinCtl 0.116s 0.111s -4.57% 1.048x
bidict 0.091s 0.087s -4.56% 1.048x
archinstall 0.262s 0.250s -4.55% 1.048x
bandersnatch 0.131s 0.125s -4.55% 1.048x
httpx-caching 0.112s 0.107s -4.36% 1.046x
janus 0.113s 0.108s -4.31% 1.045x
nox 0.108s 0.103s -4.20% 1.044x
zipp 0.089s 0.085s -4.18% 1.044x
prefect 1.709s 1.639s -4.09% 1.043x
sphinx 0.383s 0.367s -3.98% 1.041x
urllib3 0.186s 0.179s -3.97% 1.041x
python-sop 0.110s 0.106s -3.94% 1.041x
cloud-init 0.505s 0.485s -3.92% 1.041x
git-revise 0.106s 0.102s -3.88% 1.040x
speedrun.com_global_scoreboard_webapp 0.120s 0.116s -3.87% 1.040x
werkzeug 0.183s 0.176s -3.84% 1.040x
alectryon 0.085s 0.082s -3.82% 1.040x
starlette 0.131s 0.126s -3.81% 1.040x
boostedblob 0.140s 0.134s -3.80% 1.040x
paasta 0.250s 0.240s -3.79% 1.039x
dulwich 0.359s 0.346s -3.73% 1.039x
cwltool 0.230s 0.221s -3.73% 1.039x
meson 0.599s 0.577s -3.66% 1.038x
artigraph 0.164s 0.158s -3.64% 1.038x
cibuildwheel 0.116s 0.112s -3.62% 1.038x
discord.py 0.314s 0.303s -3.55% 1.037x
sockeye 0.895s 0.864s -3.52% 1.036x
CPython (cases_generator) 0.900s 0.869s -3.40% 1.035x
dacite 0.075s 0.072s -3.37% 1.035x
CPython (peg_generator) 0.103s 0.100s -3.35% 1.035x
django-modern-rest 0.161s 0.155s -3.35% 1.035x
twine 0.115s 0.111s -3.32% 1.034x
flake8 0.107s 0.103s -3.31% 1.034x
aiohttp 0.265s 0.257s -3.26% 1.034x
packaging 0.108s 0.105s -3.03% 1.031x
ppb-vector 0.087s 0.084s -2.96% 1.031x
materialize 0.436s 0.423s -2.95% 1.030x
check-jsonschema 0.099s 0.096s -2.93% 1.030x
cki-lib 0.122s 0.119s -2.89% 1.030x
mypy 1.049s 1.019s -2.87% 1.030x
porcupine 0.123s 0.120s -2.84% 1.029x
websockets 0.122s 0.119s -2.81% 1.029x
hydpy 0.598s 0.582s -2.73% 1.028x
asynq 0.105s 0.102s -2.61% 1.027x
aiohttp-devtools 0.892s 0.869s -2.60% 1.027x
com2ann 0.893s 0.870s -2.58% 1.027x
bokeh 0.313s 0.306s -2.48% 1.025x
downforeveryone 0.087s 0.084s -2.48% 1.025x
zulip 0.953s 0.930s -2.47% 1.025x
ibis 0.762s 0.744s -2.38% 1.024x
rich 0.140s 0.137s -2.34% 1.024x
dragonchain 0.128s 0.126s -2.12% 1.022x
tornado 0.208s 0.204s -2.06% 1.021x
psycopg 0.246s 0.241s -1.99% 1.020x
pybind11 0.122s 0.120s -1.87% 1.019x
python-htmlgen 0.084s 0.082s -1.77% 1.018x
sympy 3.226s 3.174s -1.62% 1.016x
dd-trace-py 1.450s 1.426s -1.61% 1.016x
scrapy 0.267s 0.263s -1.37% 1.014x
rotki 0.799s 0.789s -1.28% 1.013x
ignite 0.284s 0.281s -1.20% 1.012x
scipy-stubs 0.535s 0.529s -1.14% 1.012x
CPython (Argument Clinic) 0.880s 0.872s -0.94% 1.009x
pywin32 0.493s 0.490s -0.70% 1.007x
django-test-migrations 0.090s 0.090s -0.68% 1.007x
spack 0.709s 0.705s -0.65% 1.007x
django-stubs 0.230s 0.229s -0.61% 1.006x
scikit-build-core 0.140s 0.140s -0.45% 1.005x
pytest 0.258s 0.257s -0.41% 1.004x
openlibrary 0.273s 0.272s -0.30% 1.003x
kornia 0.250s 0.251s +0.59% 0.994x
core 3.281s 3.302s +0.64% 0.994x
mongo-python-driver 0.272s 0.275s +1.01% 0.990x
pandas 2.317s 2.345s +1.22% 0.988x
optuna 0.396s 0.415s +4.64% 0.956x
apprise 0.435s 0.457s +5.02% 0.952x
scikit-learn 1.357s 1.448s +6.74% 0.937x
scipy 3.722s 4.044s +8.64% 0.920x

@MichaReiser MichaReiser left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I suggest running the benchmarks on a few more projects to ensure we're not overfitting a certain set of projects

Comment thread scripts/build_ty_pgo.py Outdated
"--project",
str(checkout),
"--python-version",
PYTHON_VERSION,

@MichaReiser MichaReiser Aug 11, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for hardcoding the python version. Isn't this penalizing projects using older Python versions? Did you run your benchmarks with the project's real Python version?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency, but I can change it.

@MichaReiser MichaReiser left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that we don't install any third party dependencies. Is this intentional, why? It makes them very different from any real world project.

Did you run your benchmarks with third party dependencies installed?

@MichaReiser
MichaReiser self-requested a review August 11, 2026 11:41
@charliermarsh

Copy link
Copy Markdown
Member Author

No, they're omitted from training and evaluation. It's just much simpler that way.

@charliermarsh

Copy link
Copy Markdown
Member Author

I'll redo the analysis both ways.

@MichaReiser

Copy link
Copy Markdown
Member

We do have a setup_primer_project script in Ruff. That might be useful here.

I think, at least, we need to do the evaluation with dependencies installed.

@charliermarsh

Copy link
Copy Markdown
Member Author

Yup, makes sense.

@charliermarsh

Copy link
Copy Markdown
Member Author

Good news all around...

The current pipeline doesn't regress CPU time in any of the ecosystem projects. All 162 projects showed improved CPU time (20.7% lower on average), two projects showed wall-time regressions that could be noise (18.8% faster on average). All the projects that regressed in your report now improve by 10-20%.

Adding Home Assistant, or Home Assistant and SciPy, had almost no effect:

 Addition to current corpus           Average CPU improvement versus current PGO    Projects slower    Executable growth    Compressed growth
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━
 Home Assistant: 109 files                                                 0.26%           68 / 162                0.16%                0.24%
───────────────────────────────────  ────────────────────────────────────────────  ─────────────────  ───────────────────  ───────────────────
 Home Assistant + SciPy: 180 files                                         0.15%           73 / 162                0.20%                0.42%

On the dependency front, even the existing pipeline shows an enormous improvement when evaluating over "projects with dependencies installed", but it gets even better if you train with dependencies installed too:

 Comparison                                   CLI CPU    CLI wall time               Editor edits      Binary size
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━  ━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━
 Existing PGO versus no PGO                19.8% less     19.2% faster                          —     8.0% smaller
────────────────────────────────────────  ────────────  ───────────────  ─────────────────────────  ───────────────
 Dependency-trained PGO versus no PGO      20.9% less     20.3% faster               18.9% faster     8.8% smaller
────────────────────────────────────────  ────────────  ───────────────  ─────────────────────────  ───────────────
 Dependency-trained versus existing PGO    1.27% less     3.94% faster    Statistically unchanged    0.72% smaller

@MichaReiser

Copy link
Copy Markdown
Member

Sorry. I didn't mean to add homeassistant to the training corpus. Only adding it to the evolution corpus

@charliermarsh

Copy link
Copy Markdown
Member Author

Oh np. With current PGO it does improve by 18% or so IIRC from the session. I also modified the script to use the project Python version.

Comment thread scripts/build_ty_pgo.py
Comment on lines +163 to +169
EcosystemProject(
name="astropy",
repository="astropy/astropy",
revision="b779108c7cec25c840c0f744fdf2a1550441e309",
source_directories=("astropy/units",),
python_version="3.11",
),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why you aren't using setup_primer_project? It also has an option to print the ty invocation command.

Comment thread scripts/build_ty_pgo.py
Comment on lines +339 to +340
"--no-progress",
"-qq",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important, but I believe -q implies --no-progress

Comment thread scripts/build_ty_pgo.py
Comment on lines +631 to +632
try:
initialized = request(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm waiting for the day when we'll break this 😆

I, for sure, will not think of our PGO pipeline when considering breaking changes

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to setup CI for this...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be simpler in ty, because we only update ty in release builds (can it run at the same time as when we run our other release jobs?)

@charliermarsh
charliermarsh force-pushed the charlie/ty-linux-pgo-prototype branch from 88658d8 to 58f75f6 Compare August 11, 2026 19:48
@charliermarsh
charliermarsh merged commit c25c341 into main Aug 11, 2026
34 checks passed
@charliermarsh
charliermarsh deleted the charlie/ty-linux-pgo-prototype branch August 11, 2026 22:23
charliermarsh added a commit that referenced this pull request Aug 11, 2026
## Summary

This PR enables PGO for ty's macOS ARM64 releases, following the
approach outlined in #4213. The
existing linker configuration and safe identical-code folding are
preserved.

Across seven held-out macOS projects, `ty check` finishes 5.0% faster
and uses 7.1% less CPU; incremental language-server edits are 7.3%
faster. The production executable is 3.6% smaller than without PGO,
although the wheel and release archive are each 2.8% larger.

The macOS ARM64 release job uses the same Namespace runner as Ruff,
reducing its full-stack PGO build from 23m 47s to 6m 39s (72%). Three
Namespace builds finished in 6m 38s to 7m 59s; the previous Depot runner
took 11m 20s.
charliermarsh added a commit that referenced this pull request Aug 11, 2026
## Summary

This PR enables PGO for ty's Windows x86-64 releases, following the
approach outlined in #4213. Both
builds preserve static Visual C++ runtime linking.

On the Windows holdout set, `ty check` uses 19.7% less CPU and finishes
17.5% faster, while incremental language-server edits get 14.7% faster.
The release binary gets 9.6% smaller, and the wheel and release archive
each get 3.4% smaller.

Only the Windows x86-64 PGO job uses the same 16-core Namespace runner
as Ruff, reducing the full-stack build from 26m 08s to 12m 24s (53%).
The previous 32-core Depot runner took 14m 17s. With Linux x86-64 at 11m
28s and Linux ARM64 at 11m 37s, Windows is now the release bottleneck.
The remaining Windows targets keep their existing runners.
charliermarsh added a commit that referenced this pull request Aug 11, 2026
## Summary

This PR enables PGO for ty's Linux ARM64 releases, following the
approach outlined in #4213. (To
enable PGO, we also move to a native ARM64 runner.) The existing
jemalloc page-size configuration is preserved.

Across four held-out Linux ARM64 projects, `ty check` uses 15.0% less
CPU and finishes 11.9% faster. The release binary gets 4.9% smaller, the
wheel gets 1.2% smaller, and the release archive gets 1.4% smaller.

The Linux ARM64 PGO build uses an eight-core Depot runner, reducing its
runtime from 14m 51s on a four-core GitHub runner to 11m 37s (22%). In
the matched full-stack run, eight-core Linux x86-64 finished in 11m 28s
and Namespace Windows finished in 12m 24s, making Windows the release
bottleneck.
George-Ogden pushed a commit to George-Ogden/ruff that referenced this pull request Aug 16, 2026
## Summary

This PR enables PGO for Ruff releases, starting with Linux x86-64.

### Design

The release pipeline is modified as follows:

- We build an instrumented, stripped release binary.
- We run `check` and `format` on a corpus of projects from our ecosystem
reports -- specifically, eight of the pinned ecosystem projects that we
use in the ty CI. (In total, it's 502 Python and stub files.)
- We merge the profiles via `llvm-profdata`.
- We feed the result back into the existing `maturin` build.

### Results

We evaluate performance on a held-out corpus: Prefect, Django, Pandas,
scikit-learn, SciPy, and SymPy.

Results are as follows:

| Held-out project | `ruff check` | `ruff format` |
| --- | ---: | ---: |
| Django | 10.9% faster | 6.6% faster |
| pandas | 16.3% faster | 9.7% faster |
| scikit-learn | 10.8% faster | 7.9% faster |
| SciPy | 14.7% faster | 5.6% faster |
| SymPy | 17.1% faster | 11.4% faster |
| Geometric mean | **14.0% faster** | **8.3% faster** |

Beyond runtime:

- **Binary size decreased by 6.2%** (27.96 MB to 26.23 MB).
- **Release pipeline gets about 2x longer** (non-PGO release build took
7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s
optimized wheel).

### Stack

Additional platforms are covered in subsequent PRs in the stacked;
platforms that are lower-priority at at-all difficult to run on natively
are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS
ARM, and Windows x86-64.

I also attempted BOLT in astral-sh#27588,
but I've decided against pursuing that for now; see the results in that
PR which speak for themselves.

ty and uv will follow the same approach; see the stacks here:

- astral-sh/ty#4213
- astral-sh/uv#21001

See: astral-sh#7055.
charliermarsh added a commit to astral-sh/uv that referenced this pull request Aug 17, 2026
## Summary

This PR enables PGO for uv releases, starting with Linux x86-64.

### Design

The release pipeline is modified as follows:

- We build instrumented `uv` and `uvx` release binaries inside the
existing manylinux container.
- We train directly on eleven shared `test/ecosystem` fixtures:
cibuildwheel, Cookiecutter, Flask, HTTPX, LLM, the OpenAI Python SDK,
Poetry, pytest-cov, Sentry, Zulip, and the 38-project pyx workspace.
- Every project exercises cold and warm `uv pip compile` resolution,
cold and warm `uv lock` resolution, `uv export`, `uv pip` installation,
and `uv sync` installation on every release platform.
- Per-project `exclude-dependencies` settings omit packages without
compatible release-platform wheels while preserving the rest of each
real dependency graph.
- Raw profiles are merged online by workload family, then merged with
`llvm-profdata` and fed back into the existing `maturin` build.
- JupyterLab, Saleor, Semantic Kernel, Transformers, and Warehouse
remain separate, held-out evaluation projects.

### Results

We benchmarked the builds on Linux x86-64 using the expanded corpus for
training and five separate held-out ecosystem projects for evaluation.

| Held-out project | Dependency resolution | Wheel installation |
Project locking | Lockfile export |
| ---------------- | --------------------: | -----------------: |
---------------: | --------------: |
| JupyterLab | 11.2% faster | 0.0% | 10.1% faster | 10.6% faster |
| Semantic Kernel | 9.3% faster | 7.1% faster | 12.5% faster | 8.7%
faster |
| Transformers | 11.7% faster | 4.1% faster | 9.5% faster | 11.0% faster
|
| Saleor | — | — | 9.7% faster | 10.0% faster |
| Warehouse | — | — | 9.7% faster | 7.6% faster |
| Geometric mean | **10.7% faster** | **3.7% faster** | **10.3% faster**
| **9.6% faster** |

Across all sixteen held-out workloads, wall time decreased by **8.9%**
(95% CI: 8.4–9.8%) and CPU time decreased by **8.4%** (95% CI:
7.9–9.0%). The matched Linux executable was **13.3% smaller** (58.91 MB
to 51.05 MB), or **12.3% smaller** after gzip compression (22.68 MB to
19.90 MB).

(Saleor and Warehouse include source-only dependencies, so they are
benchmarked on locking and export rather than wheel-only installation.)

### Stack

Additional platforms are covered in subsequent PRs in the stack: macOS
ARM64, Windows x86-64, and Linux ARM64.

Ruff and ty follow the same approach; see the stacks here:

- astral-sh/ruff#27570
- astral-sh/ty#4213
VedantMadane added a commit to VedantMadane/uv that referenced this pull request Sep 21, 2026
This PR enables PGO for uv releases, starting with Linux x86-64.

The release pipeline is modified as follows:

- We build instrumented `uv` and `uvx` release binaries inside the
existing manylinux container.
- We train directly on eleven shared `test/ecosystem` fixtures:
cibuildwheel, Cookiecutter, Flask, HTTPX, LLM, the OpenAI Python SDK,
Poetry, pytest-cov, Sentry, Zulip, and the 38-project pyx workspace.
- Every project exercises cold and warm `uv pip compile` resolution,
cold and warm `uv lock` resolution, `uv export`, `uv pip` installation,
and `uv sync` installation on every release platform.
- Per-project `exclude-dependencies` settings omit packages without
compatible release-platform wheels while preserving the rest of each
real dependency graph.
- Raw profiles are merged online by workload family, then merged with
`llvm-profdata` and fed back into the existing `maturin` build.
- JupyterLab, Saleor, Semantic Kernel, Transformers, and Warehouse
remain separate, held-out evaluation projects.

We benchmarked the builds on Linux x86-64 using the expanded corpus for
training and five separate held-out ecosystem projects for evaluation.

| Held-out project | Dependency resolution | Wheel installation |
Project locking | Lockfile export |
| ---------------- | --------------------: | -----------------: |
---------------: | --------------: |
| JupyterLab | 11.2% faster | 0.0% | 10.1% faster | 10.6% faster |
| Semantic Kernel | 9.3% faster | 7.1% faster | 12.5% faster | 8.7%
faster |
| Transformers | 11.7% faster | 4.1% faster | 9.5% faster | 11.0% faster
|
| Saleor | — | — | 9.7% faster | 10.0% faster |
| Warehouse | — | — | 9.7% faster | 7.6% faster |
| Geometric mean | **10.7% faster** | **3.7% faster** | **10.3% faster**
| **9.6% faster** |

Across all sixteen held-out workloads, wall time decreased by **8.9%**
(95% CI: 8.4–9.8%) and CPU time decreased by **8.4%** (95% CI:
7.9–9.0%). The matched Linux executable was **13.3% smaller** (58.91 MB
to 51.05 MB), or **12.3% smaller** after gzip compression (22.68 MB to
19.90 MB).

(Saleor and Warehouse include source-only dependencies, so they are
benchmarked on locking and export rather than wheel-only installation.)

Additional platforms are covered in subsequent PRs in the stack: macOS
ARM64, Windows x86-64, and Linux ARM64.

Ruff and ty follow the same approach; see the stacks here:

- astral-sh/ruff#27570
- astral-sh/ty#4213
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release Related to the release process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants