This request is not even Python-specific. For example, one can imagine being able to set tailor-related options in different parts of the source tree.

I've created an issue about it #20898

Is this line of thinking stuff like "run different versions of black"?

I have a clear example in which having different resolves in a monorepo but having only one pytest setting version is causing trouble. It would be ideal to pass on a pydentic resolve through the BUILD file for example.

This idea seems distinct from a new Python backend, although a new Python backend could be written to create synthetic top-level targets based on detecting Python code.

One concern is how users will be able to understand what the new backend has inferred. Debugging magic inference is not fun ...

Well, I want to at least experiment with not modeling targets as expansively as we do today, even internally.

One concern is how users will be able to understand what the new backend has inferred.

Isn't something that could be done simply with the existing previewing goals such as pants list or pants peek?

Just so I understand the idea, we are talking about omitting BUILD files in folders and just infer what would have been the tailored build file instead? In some way, this is no different than generatives target such as python_sources that infer multiple python_source?

I think the nudge I gave @kaos did a lot of the groundwork here. Allow for a root to say "I am a Python (or whatever) root." Then at the various directory levels you can override the default values. Targets are referred to as files.

yea, I have some half done works and ideas on this topic. But it doesn't get rid of the targets as a concept.. merely adds support for not having to present all the scaffolding in actual BUILD files.

That seems wise.

Side note: This change could be made in the existing backend in a straightforward manner (i.e., rename python_tests to python_pytest_tests etc.) We could even then introduce a unittest test runner (e.g., python_unittest_tests).

As a nitpick, per implementation targets feels gross. I get why we need them, but blah from the API side. I also know we do it in a lot of places already, but... again... blah.

Something we have in the JS backend (I think it's still there, anyways) is a field that lets you pick your implementation/runner for package management via a field (npm, pnpm, yarn). That may be at the NodeJS subsystem level though, I'd need to look at it again.

Would be nice at least for the generators, to reduce the number of concepts new (and honestly, existing) users would need to learn.

python_tests(
  name="",
  runner="nose",
)

The per-implementation field details would be where things fall off the rails. I suppose we could LSP away a lot of the complexity, and then generators could generate a per-implementation variant - copying the associated field data (which would need to have a dict or kwargs or something).

python_tests(
  name="",
  runner="some_rust_runner",
  dependencies=[...],
  rust_version="1.78",
  ...
)

# generates

python_some_rust_runner_test(
  ...
)

I'm entirely handwaving all technical complexity here :)

As a nitpick, per implementation targets feels gross. I get why we need them, but blah from the API side. I also know we do it in a lot of places already, but... again... blah.

I agree. The per-framework target types almost seem like something which should belong in an "intermediate" build graph. Pants can infer there are tests and build an internal representation of the technologies used. Even python_tests feels too intermediate. Which begs the question: What is the truly high-level config language for builds?

+1 for abstracting away pex, in some cases it might beneficial to use a virtual venv as the execution context instead

I think this could probably just be implemented in Pex using the pip --fast-deps flag perhaps?

Can the slow-down be attributed to a lack of built wheels for certain dependencies?

I was going to play with integrating uv. The main reason we haven't done this in the existing backend is that it can't generate multiplatform lockfiles, but maybe we switch to having separate lockfiles per platform. See pex-tool/pex#2371 for more info.

On a related note is to not invalidate the entire world on every change to the lockfile/requirements. This PR is a first step to support tracking affected targets in this case.

Slow lockfile generation will likely cause us not to adopt Pants unfortunately, so using something like uv would be incredible. I realize that this is a bit of apples and oranges, but: It takes 4 minutes on a decent cloud-based machine to generate a lockfile, even with everything fully cached. It takes another 2 minutes to export the environment.
Paying a 6 minute penalty to bump a module version is going to be hard for our team to swallow.

In my experiments with slow generate-lockfiles, even on a fully-cached run (where I had JUST run generate-lockfiles and made no changes), even the 2.21.0a branch, 3 minutes was simply spent in pip checking pypi, realizing that the version it already had were current, and then another 10s copying some wheels. There are likely some settings I'm missing to speed this up.

I've got a gist of the pip download log here: https://gist.github.com/Taytay/492c12eaedce6c7999e0028fb6a9a50a
Slack post regarding this behavior here: https://pantsbuild.slack.com/archives/C046T6T9U/p1715862601075849

Do you mean seamless interop between Rust and Python rule code? Idea: an out-of-tree plugin contains both Python rule code and Rust intrinsics and/or support code callable from Python

Idea: an out-of-tree plugin contains both Python rule code and Rust intrinsics and/or support code callable from Python

[sharpens pencil, opens notepad]
...go on... 😄

Specifically on the out-of-tree part? Why would it be handled any differently from today?

Don't get me wrong, I've been kinda pitching pushing backends out-of-mainline for a while (with no expectation that can happen until we have API stability)

The thought is a very nascent one: using this new Python backend as an example, imagine if it could be developed out of tree with both its Python rule code and the Rust-based dependency inference code living together in the same repository. And they would be packaged together and consumable by users by just referencing the name of this published plugin in pants.toml.

Note: I am intentionally glossing over how it would be distributed (whether on PyPi, pexes, whatever.) and how it would be packaged.

What matters is that:

1. We have an API which allows both Python and Rust code to plug into the Pants core.
2. There is a packaging mechanism which allows Python and Rust to live together separately from the Pants core.
3. Some way (other than PyPi?) for discovering and distributing these plugins.

Hmm, that's interesting... I think I need to fiddle around with pyo3/maturin a bit more with in-repo plugins - might help at least refine the proposition.

Do you know if the slow-down is due to some amount of overhead in initializing PyTorch? Are there other causes?

@tdyas : Since this behavior isn't seen outside of pants (that I'm aware of), this is likely due to the sheer size of the pytorch dependency, and likely some pex-related dependency movement/detection, right? I thought some unpacking needed to happen in a case of pants run

For the latter, there's ongoing work for this here #20364

For the former (which is something I was going to "wishlist" myself), I think #20531 could solve it.

I still think the solution to this is "edge metadata" (as @stuhood called it). E.g. allow the user to say how a dependency ought to exist.

I guess this is basically the same as #20897 (comment), except applied to external tools instead of Pants itself.

I like the sound of this, but it seems to inherently conflict with #20897 (comment) which I also like. There are two philosophies, and I'm not sure which I subscribe to:

1. Pants should be a super thin wrapper which transparently delegates to language-specific tools and preserves as much of their functionality as possible
2. Pants should have a single engine which gives a consistent cross-language experience, and only use language-specific tools to map each language onto the interface

I can see each approach being beneficial in different scenarios:

• Right now, I'm adding pants to a python-only repository. It's frustrating that pants has so much "bespoke" functionality, so I can't interchange between pants and other tools. I almost want pants to just be a plugin for faster tests, with a config section in pyproject.toml
• On the flip side, if I was adding pants to a repo which had multiple languages (i.e. java and python), I might find it really frustrating that dependency resolution logic or syntax is different (i.e. maven vs poetry), and it would be nicer if there was a consistent Pants abstraction across both

(I'm assuming a Python code base..)
Have you tried using https://www.pantsbuild.org/2.20/reference/targets/python_tests#batch_compatibility_tag and perhaps also bump https://www.pantsbuild.org/2.20/reference/goals/test#batch_size

I imagine that you would end up with a single batch (i.e. one pytest invocation with all files) if all python_test targets share the same compatibility tag, and the batch size is large enough to hold them all. Hint: use __defaults__ to set the compat tag on all targets in a subtree using:

__defaults__({python_tests: dict(batch_compatibility_tag="our-compat-tag")})

https://www.pantsbuild.org/2.20/docs/using-pants/key-concepts/targets-and-build-files#field-default-values

If we did a single batch, wouldn't we lose all caching?

yea, it would trade the parallelism for caching.. so not ideal from that perspective.

This may require some amount of rethinking how Pants caches build results then. [For the benefit of other readers:] Right now, the REAPI model used by Pants enforces using process invocations as the basis of caching and caches using all fields of the process invocation as the cache key (since the Command and Action REAPI structs are hashed). Moreover, those process invocations (if not in the cache) are expected to produce the expected output. If we are going to relax that constraint, then we are no longer in REAPI's data model (which may be fine but it will need thought).

An idea of mine from a while ago: We might work around the REAPI model's constraints by constructing "synthetic" process invocations for which we just upload results which were otherwise computed in a batch via a single process invocation. The synthetic process should still conceivably invoke a process which could recompute just that file's result. This could conceivably work for linting and formatting process invocations.

Holy crap, this is an incredible write up. It should absolutely be a discussion all of its own - as a lot of these concerns aren't Python specific, but Pants and/or new-user specific.

Just going to touch on a couple of points.

1. Learning curve

I think better (read: less thorough, but easier to digest) documentation would help. We've talked about a "recipes" kinda page - but yeah, something more like what I put up here https://tldr.inbrowser.app/pages/common/pants would be useful for the first 10 minutes of a user's Pants workflow.

I would also like if pants itself guided users somehow with trivial suggestions (no idea how this could work or what it would look like).

A split between "starting a new project" and "using an existing project" might also be a good approach.

Documentation suggestions always welcome!!!

2. No official VSCode extension

Correct, not yet. I wrote Suspenders - and it was on hold while I was pushing through some pants changes that were required to make Suspenders useful. I'm slowly going through the process of adding code lenses, test runners, and LSPs (and unit testing!) - in the next 1-2 months I'm hoping for a decently featured, but still "preview" release.

The BUILD LSP it already a huge win for me locally, but I need to make it tailor itself to each project, rather than just my projects :)

4. Discoverability of goals seemed harder than something like a makefile or a package.json that exposes

Compared to package.json, yeah - less sure about the makefile comparison 😆

This could be an interesting usage of alias, where each team can carve out its set of scripts, but that would be at a monorepo level currently - rather than a per project level (I think, I've never tried it otherwise).

I personally suggest that to teams, but it's up to them to use it. It's cool though, because it stretches out how long a team can go before looking at the docs (which is great for new users).

For instance, one of the companies I set it up for - my readme (and aliases) are all they need for the most part - the fact that they call pants is an almost an afterthought, could just as well be pnpm or cargo to them.

6. Necessity of a BUILD file in each sub-directory of my modules

You don't need to and I never do it. Gives me flashbacks of my older CMake days.

I personally have a single BUILD file at the top-level of each of my sub-projects (e.g. https://github.com/sureshjoshi/pants-plugins/tree/main/examples/python/hellofastapi has sources="**/*.py")

Tailor automatically generates one per location of sources, by default I think.

7. No built-in templating for creating a new module/project in my monorepo

Yeah, this is a tricky one because I haven't seen two companies with the same ideas of what a monorepo should look like. Pants prefers to be indifferent to that aspect, so long as BUILD files are present.

Not sure what the solution would be here - other than some cookie cutters that you could init against (ala yeoman)

9/10: Speed

Yeah, there is never a "too fast" :)

Each level of tooling on top of no tooling adds overhead. Personally, speed is my biggest issue with Pants too - but in my case, it's daemon init while working on Pants itself. There's a lot of work going into this side, but I'd agree that we want to optimize the most often called workflows.

Just use this backend, and we'll bootstrap Python for you, handle linting, dependency installation/subset detection, deployment to common targets, etc

I wonder if an interactive tailor might be the partial-solution to this kinda thing. Instead of doing something and having new users read the docs on how to configure it, if it inferred a lot of information and asked clarification questions instead.

There's a lot of work to get to what you described, but making life easier for new users is always a good thing.

TL;DR: I'd love for the next Python backend to be faster and simpler for the common case :)

Yes, yes, a thousand times yes.

Thanks @sureshjoshi ! And thanks for Suspenders!
Yes, even that short documentation helps a LOT! : https://tldr.inbrowser.app/pages/common/pants

Fair point about makefiles being obtuse! I just meant that they expose named targets. :)

You don't need to and I never do it. Gives me flashbacks of my older CMake days.
I personally have a single BUILD file at the top-level of each of my sub-projects (e.g. https://github.com/sureshjoshi/pants-plugins/tree/main/examples/python/hellofastapi has sources="**/*.py")

Oh thank goodness! I think tailor is just eager. :)

Yeah, this is a tricky one because I haven't seen two companies with the same ideas of what a monorepo should look like. Pants prefers to be indifferent to that aspect, so long as BUILD files are present.
Not sure what the solution would be here - other than some cookie cutters that you could init against (ala yeoman)

Oh! That makes sense! I don't know enough to have opinions yet. :) I think I was looking for something a bit more opinionated and hand-holding in that regard. I think the flexibility is awesome in the long run though.

I wonder if an interactive tailor might be the partial-solution to this kinda thing. Instead of doing something and having new users read the docs on how to configure it, if it inferred a lot of information and asked clarification questions instead.

I really like that idea. Of course it would be difficult to make it flexible enough for "ALL" existing repo scenarios, but questions like the following would have helped me tremendously in adopting it for my repo, even if it didn't move anything for me. Of course, they would help with a new repo too...Even if my repo isn't set up correctly at first, knowing that I need to move my code into the 'src' folder after I'm done is super helpful!
"Q: What version of Python do you want to be your default? If you are unsure, just leave it blank. [3.12] : "
"Q: Where will each of your python project folders be located? This is called a "source root". (We recommend that they go into 'src' or 'src/python'.) Don't worry - if there are more locations, you can specify those in a moment. [src/python]: "
"Q: "
"Q: Would you like for me to create two sample projects in 'src/python' that show you some basic concepts? Y/n: "
"Q: Would you like me to create sample Github Action showing how to build and package those sample projects? Y/n: "
"Q: Would you like me to add a section to your readme with common pants commands?"
"Q: What IDE do you use? [VSCode, PyCharm, Other...]"
"Q: Which formatter would you like to use?..."
Summary: [Here's what we did. Here are the common commands you will need to know... Next Steps: ...]

I personally have a single BUILD file at the top-level of each of my sub-projects (e.g. https://github.com/sureshjoshi/pants-plugins/tree/main/examples/python/hellofastapi has sources="**/*.py")

Oh wow I was explicitly avoiding this because I thought pants did "something" at the target-level, like dependency graphing or cache invalidation. Are you saying there's currently no downside to declaring top-level targets?

It's nuanced, but those python_sources() "targets" are actually "target generators" - they are macros that expand to a python_source() target for each underlying file. So there is no downside to having a single top-level target. It used to be that if you wanted to override some field value for a specific file (say a timeout for a test) you had to create a new target for just that file, which meant you had to exclude the file from the sources of the main target, which was a hassle. But now that you can use overrides on that main target, even that reason is moot.

@jasondamour Benjy just sniped the answer from me, but higher level, still feeling the sting from years of CMake and even if there were small downsides - I would pay that willingly to not have hundreds of identically named files 😆

Automatically fetch a python interpreter (similar to how rye automatically manages your toolchain from https://github.com/indygreg/python-build-standalone)

This is funny, because I thought we DID do this (circa this post https://www.pantsbuild.org/blog/2023/03/31/two-hermetic-pythons). scie-pants (aka pants fetches the appropriate python-build-standalone to run itself) and I thought there was a feature which allowed picking a PBS, or grabbing one based on interpreter constraints. I might have conflated it with pyenv though (or just Docker builds: https://www.pantsbuild.org/2.20/reference/subsystems/python-bootstrap#internal_python_build_standalone_info)

I'm a +1 for this being automatic, if it isn't.

Oh, also, I don't think this requires a new Python backend - this should be possible (again, if we're not already able to do it) in the current backend.

Thanks for this "talk"! It's very very helpful. Particularly since it dovetails with much of my own thinking (getting rid of BUILD files unless needed, leaning in to uv even if it means needing per-platform lockfiles, simplifying the UI, etc etc). The current Python support was strongly influenced by how some specific companies managed their monorepos, so getting a more diverse set of opinions is super helpful.

Honestly avoiding BUILD files scares me a bit. I thought that it was a nice way to structure it: "BUILD files mean to BUILD this folder"...
But that's for someone who is adopting it progressively, so yeah, maybe I'll feel better later. There's a tension between easy/hidden "magic" and explicit repetitive determinism. I thought the BUILD files with very few lines of "code" were a nice compromise, but maybe with the other python-related files (project.toml?) they are repetitive.

The BUILD files get created everywhere when you run pants tailor :: which you probably should run in your CI to make sure you didn't forget it anyway, so there are no special semantic such as "If there's no BUILD file in this folder don't build it" other than "I forgot to run pants tailor". On top of that the BUILD file are 99% of the time just a quick magic spell like python_sources(). It doesn't make anything explicit, I already can see there are .py in this folder. If anything they are annoying when you cleanup some folder such as removing a conftest.py and now pants complains about a target with no source.

But YES! Pulling down a python was something I was 1) surprised didn't happen by default, and 2) was slower than I expected after I enabled pyenv. :) I'd love it if a new dev just needed to run pants and would be off to the races...

Exactly this!! #21194

Should the new backend should ditch pex as the underlying venv setup technology?

Would it be easier to just have Pants build venv's directly?

I ended up adding yet another special dependencies field in #21062 to add this to the existing python backend.

The new python backend should have a richer dependencies field that allows edge tagging/annotation to describe how a dependency is required (as a wheel file? As an editable install? As a fully installed wheel? As a runtime package dependency? As a file? As a resource?) or in this case, which entry points are required.

The purpose here is to reimagine Python support without the conceptual burden of all the legacy cruft, based on the many lessons we've learned in the last 4-5 years. That seems very hard to achieve via evolution.

Maybe evolution is the wrong way to describe what I am suggesting. The thought is to do breaking changes in a separate repository ("channel"), get it tested, stabilize it, and then merge it into main for release as the next iteration of the Python backend. Then repeat that process again and again.

Let's find a way to develop breaking changes in a way that does not break main nor regular Pants releases. I believe sequencing a number of smaller stabliized breaking changes over time will end up better for users and for Pants developers. Users will always have a working Pants release with Python support. As developers, we won't have to spend inordinate amounts of time building out a fully featured rewritten backend before we consider having users port to it. (And we would need to do that because as a user I am less likely to use a new backend which does not support everything I need out of it.)

We have already seen this issue with Pants v2 over Pants v1. While Pants v2 probably required being a rewrite, even to this day Pants v2 still does not support the full breadth of the user-facing features that Pants v1 did (outside of Python). I would set a high bar for treating this as a full rewrite versus a sequence of breaking changes developed to the side.

As a user who is excited about improvements to the Python backend, but is unaware of any tradeoffs, I would personally prefer an incremental approach where improvements can be shipped sooner, rather than waiting for a full rewrite before seeing improvements.

Can you give some stats on how many targets/files/etc you have? For a cold run, sure, but even for warm/hot runs it takes that long?

I also experience this and it's extremely painful

Here's the SCC result

$ scc .
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Python                   61481  11128150   816441    321013  9990696     418941
HTML                     21527   4603162   858670    788851  2955641          0
JSON                     10798   8408038      482         0  8407556          0
Plain Text                4385    210323    22270         0   188053          0
Markdown                   885     66991    16496         0    50495          0
CSV                        567   1832170      149         0  1832021          0
XML                        513    104085       19         4   104062          0
XML Schema                 362     77045     2929         0    74116          0
YAML                       291     35289      475       356    34458          0
Shell                      289     14743     2044      1492    11207       1630
Smarty Template             62      2702      353         0     2349        387
SQL                         45     38095      319       121    37655        131
INI                         37       999      176       167      656          0
TOML                        35      5544      112        60     5372          3
ReStructuredText            30      2854      912         0     1942          0
JavaScript                  21     13040      119        93    12828        160
gitignore                   16       339       59        57      223          0
BASH                        10      1060      124       112      824        148
CSS                          8      1469      259         2     1208          0
Expect                       7        85        2         0       83         11
Properties File              7        67        6         6       55          0
CloudFormation (YAM…         6       178        7        24      147          0
Dockerfile                   6        86       17         4       65         10
Mako                         3        72       21         0       51          0
Stata                        3        45        0         0       45          0
Makefile                     2        43       11         7       25          6
SVG                          2        41        1         1       39          0
Web Services Descri…         2      1600        0        27     1573          0
Docker ignore                1        23        0         0       23          0
Jinja                        1        50        0         0       50         16
TypeScript                   1       264       41        59      164         30
───────────────────────────────────────────────────────────────────────────────
Total                   101403  26548652  1722514   1112456 23713682     421473
───────────────────────────────────────────────────────────────────────────────

Pants being so slow and resource intensive it's becoming unusable. If there's a pants command I can run to provide number of targets lmk.

To be clear, both cold run and warm runs are slow, similar to what's described here #18911

We've been running pants for a little bit and this is one of the main feedback from the developers.

We tought about using pants run as our driver, but even with warm shared cache between containers, this has been a very bad idea. Booting up the pants scheduler alone takes between 10-20 seconds, and you can't use the daemon for more than one command at a time.

For now we are backtracking on using pants run and instead using the venv from pants export.

@kuza55 How often do you find yourself generating lockfiles? Is it a frequent part of the workflow - or an occasional pain?

I've said it elsewhere in this discussion, but I'd take improved performance over basically anything else - as I'm a very impatient person (apparently).

There are a few changes coming down which should hopefully help a little, but any stats or backfilling of what packages you use that cause the most grief would also be helpful (maybe we should have a per-step tracking ticket/discussion where people could flood their requirements.txt and benchmarks).

We tought about using pants run as our driver, but even with warm shared cache between containers, this has been a very bad idea. Booting up the pants scheduler alone takes between 10-20 seconds, and you can't use the daemon for more than one command at a time.

@isra17 Two things you said feel concerning already:

1. The scheduler taking 10-20 seconds is something we run into on pants mainline, but I have a ticket pointing out that a couple of backends are doing some things they shouldn't, and we can bring that down to 6 seconds on my machine (from 20-30). I've never experienced anything remotely close to that as a consumer of pants though. I'll create a tracking ticket for this when I'm back at my computer later today
2. "Can't use the daemon for more than one command at a time" - this feels like something janky in the config or setup, could you create a ticket for this (if you already haven't)? This is not a thing that should happen.

I assume "more than one command at a time" means "more than one run of Pants at a time"? You can run multiple goals in a single Pants invocation, but they will run serially.

Genrate-lockfiles slowness impacts are not distributed evenly. If you're trying to do something new and need a new dep, or you're trying several different libraries, etc, it can be really painful, I think it regularly takes like 10 minutes for a single invocation.

This also hurts pants beginners more because they don't understand when they need to run this, and that they should basically avoid this if they can, so they get stuck running this command more than they need to.

I didn't mention it here since I thought it could get resolved separately from this, but multiple pants invocations at once is a big concern for me too. Mostly because of long running test & run targets, i.e. #20642

I also have a feeling that some other performance issues I experience are downstream of using PANTS_CONCURRENT to workaround that issue.

Tracking ticket for generate-lockfiles slowness. There are a large number of possible solutions/improvements floating around, would be good to have some canonical repos to test against:

#21223

That's not really related to this backend wishlist.

pyink can be added as another formatter backend already - in the same way we have ruff and black: https://www.pantsbuild.org/2.21/docs/writing-plugins/common-plugin-tasks/add-a-formatter