Repository navigation
Reconsider how editing environments works #886
Description
Context
Currently editing environment:
- creates a new build directory
- swaps the symlink
This means that autoreloading does not work. For example, when using with Jupyter/IPython:
- kernelspec needs to be reloaded hence need for back-and-forth switching between environments to see changes applied; in my experience this wastes about 3 minutes per each change in an environment (compared to workflow without conda-store)
- newly installed packages do not become available in a running kernel which means that possibly hours of computation may be lost as the kernel needs to be restarted to pickup smallest change in the env
- changed packages cannot be picked up by the IPython autoreloader either
If instead it worked like:
- copy existing build to a new folder
- the copied folder becomes the archivial build version
- update the environment:
- ideally by updating in place rather than rebuilding from scratch
- if needs be by rebuilding (which will still work but likely change some versions inadvertently if new versions were released, which is another issue)
- there is no need to swap the symlink so autoreloading and everything else works!
Value and/or benefit
Many minutes to hours in productivity gained (or rather not lost) for the use case of interactive environment creation by a senior data analyst.
Anything else?
No response
Activity
krassowski commentedon Sep 19, 2024
@kcpevey mentioned to me that this may be a foot gun for shared environments:
I somewhat agree, but ultimately if shared env is changed by someone else, activating it after the change will cause the same issue.
And questions the UX for user awareness:
Here I would mention that auto-reloading is not enabled by default, and users who enable it know what they are doing. Also, rebuilding and env should not take 20 minutes (but it does). I do however, agree that a notification that an environment building has completed should be shown when conda-store is used with JupyterLab, which is tracked in:
dharhas commentedon Sep 19, 2024
As an fyi, historically,
updating in place rather than rebuilding from scratchhas been a really bad idea and has ended up with folks having non-reproducible bespoke / broken environments because to recreate the environment you have to recreate every update step and that is not tracked anywhere.dharhas commentedon Sep 19, 2024
But this does go with another discussion I had had about the packaging at pycon, we actually have multiple target audiences (devs, end users etc) for environment management and we are using the same tools for all of them.
dharhas commentedon Sep 19, 2024
Is this actually a valid use case? How reliably does it work? For pure python packages maybe. To me it seems if you change the underlying environment all bets are off on whether your python objects are even valid if an install changed something under the hood. Seems like a better option would be to make sure you serialized your results.
krassowski commentedon Sep 20, 2024
Yes, in IPython installation/updating of packages via
%pipand%condamagics is supported and valid use case. These magics warn that for some packages restarting kernel may be required, but when autoreload is on it is rarely the case (it is the case for updating non-pure Python).Very well in my experience. And Databricks considers it a valid use case too, they are contributing enhancements in ipython/ipython#14500.
This is my call as an experienced user to make. I can tell if I will need to restart the kernel or not, and I often know what specific changes will be made. It is not for updating numpy from 1.x to 2.x, it is for grabbing patches for very specific bugfixes.
No, I disagree here. It sounds like blaming user here but in fact even with best serialization and caching, there are operations that always take time like loading up large files or training small/medium models. I don't think that using conda-store should be incompatible with data scientists analysing big data or training baseline/statistical models in notebooks. But maybe I misunderstand the target audience of conda-store.
krassowski commentedon Sep 20, 2024
On the other hand, the current conda-store approach leads to broken notebooks for data scientists who are not used to working with conda-store:
Why not give advanced users a choice on whether to update in place or not? If the old environment is copied as an archival build that has 0 risk, right?
Thinking about it, what I really miss is:
krassowski commentedon Sep 20, 2024
Part of the delay is that even after environment is built we need to wait for up to a minute for it to be refreshed on the
nb_conda_kernelsside: https://github.com/anaconda/nb_conda_kernels/blob/04c5fc605c08a4ced0cc45d2a6507dea40897600/nb_conda_kernels/manager.py#L18So I as a user keep restarting the environment until it clicks. If my new/edited dependency is used lower in the notebook I can waste many minutes there.
For the interactive use case we need to somehow rewrite
nb_conda_kernelsto watch kernelspecs changes on disk, or emit an event from conda-store and makenb_conda_kernelsrefresh. Interaction withnb_conda_kernelsappears in scope as this is included in the Dockerfile:conda-store/conda-store/Dockerfile
Line 34 in bde7fe4
peytondmurray commentedon Oct 3, 2024
Part of the goal of the 2024
conda-storeroadmap is to eliminate some of the painfully slow tasks that users regularly encounter, and from the description this seems like a major UX annoyance. From what I can tell this is particularly irritating becauseSo it's a combination of slow iteration and not enough feedback to the user. It sounds like there are downsides to both symlinking and update-in-place. However I feel like I don't have the full context, especially with regards to:
In
conda-store, each build is a separate specification with a corresponding lockfile, though. The idea of adding a new package to an existing environment with aconda install <package>doesn't apply. Or am I missing something?This really sounds like a problem that could be fixed by notifying the user. We could also possibly give them the option to stay on the old build or bump their own build to the latest version as well, although if we do opt for hot-reloading by eliminating the symlinking mechanism, users who stick with the old build would need to be the ones who reload (to target the old build)?
krassowski commentedon Oct 24, 2024
I agree with your summary. Just one more thing:
How can a user achieve the closest possible thing to "add a new package to environment without updating changing dependencies of anything else unless necessary", like in
conda install <package>. Is my only choice to manually add pins for every single of ~30 packages that I have in "Requested packages" section?As I user I now have a fear of adding anything to an environment (but I have to!). What is the safe path?
peytondmurray commentedon Oct 25, 2024
With conda-store you can't do this at the moment because the environment gets re-solved when a new specification is submitted. This comes from previous experiences with incremental updates:
I have trouble understanding how we'd be able to reproduce the environment you'd find yourself with if there were an option to add a new package without changing dependencies of anything else unless necessary; isn't this what happens when you
pip install <package>?pipdownloads the requested version of your package, and if the currently installed dependencies meet the requirements of the package you're trying to install, those dependencies don't change. The only way you'd be able to get back to your particular environment would be either byBut maybe I'm missing something or there's another way to do this?
About user-facing messaging: are we currently passing messages to the
jupyterlab-conda-storeextension? If not, it sounds like we'll need to do so in order to fix the messaging/notification part of this issue.krassowski commentedon Oct 25, 2024
Well, by having a lock file/
pip freezeoutput committed. Right now conda-store does not solve the problem, just shifts it. Instead of pinning every dependency in a freeze/lock file I have to pin it in the specification file.The conda-store approach might be fine, but if the only sane way to add a new package is to have everything pinned, then I think it should have a button to populate pins for all packages in spec from the currently installed versions.