Better integration with conda/conda-forge for building packages

@wolfv

The idea to rely on conda-forge for building Python packages to WebAssembly has been mentioned for a while now (#38 (comment), conda/conda#7619, conda-forge/conda-forge-bot#1052 (comment)), and in this issue I wanted start a discussion about current situation and existing challenges to move in that direction from the perspective of pyodide, as I understand it (please correct me if needed).

First the main motivation is that the present way of building all the packages in one repo is not sustainable with the increase of the number of packages and the associated increase in CI time. To resolve this we would need significant development resources, which we don't have. Even if we did, it would amount to doing many things (including a community) that already exist and work great at conda-forge, which wouldn't make sense.

Now as to challenges (it's a long post),

1. Updating emscripten

With a single repo it's relatively fast to rebuild all packages with a different version of the emsdk toolchain (emscripten, binaryen, ..) or different options. We currently still have a couple of patches applied to emscripten, and we also ideally need to update emscripten frequently to benefit from improvements and fixes (currently 1.5 years late with respect to the latest release, unfortunately). In conda-forge rebuilding all the packages with a new compiler would take longer (though this got better recently conda-forge/conda-forge-bot#1052 (comment)). Also the use-case where a) we update emscripten version b) some package fails to build c) we have to go back and change some global emscripten settings would really be unpractical I think.

This would hopefully become less of an issue with time as emscripten becomes more and more stable, but it's still an issue now (see e.g. #480 (comment))

2. Build approach

The cross-compilation of scientific Python packages (based on distutils) is difficult (scipy/scipy#8571, numpy/numpy#17620) as far as I understand, even on Linux between different architectures.

I'm not sure if this was the reason, but pyodide doesn't do cross-compilation in the classical sense. Instead it compiles the package with the host compilers, stores a log of all executed compilation commands and re-run those commands with the emscripten compiler.

3. Shared package specifications

Package specifications where chosen as close as possible to the meta.yaml in conda, and hopefully soon the package index will also use the same format (#791)

4. Artifacts format

Currently each package consist of 2 separate (.data, .js) files which we distribute via jsDelivr. Those would probably not fit as conda artifacts, which would mean that we likely need to handle some of this in any case.

5. Dependency resolution in the browser

There are two use cases for pyodide,

interactive (notebooks, etc) where having a dependency resolver in the browser (e.g. mamba) would be great.
python applications, where dependencies are known in advance, and we certainly don't want to do dependency resolution at each page load. There having a precomputed list of packages is likely the way to go.

Either way we also need to install pure python wheels (from PyPi or other custom location), so we still have this duality between conda/pyodide packages and Python wheels as well. Meaning we have to maintain a minimalistic pip (micropip) in pyodide.

I haven't followed close WebAssembly related developments at conda-forge, maybe I am missing something.

cc @wolfv @jakirkham

We have cross compilation support in conda-forge. We cross compiled numpy, scipy, matplotlib for osx-arm64 platform. First thing to do would be to add a platform to conda and conda-build. something like wasm-32.

Thanks for writing this up Roman! 😄

This sounds like a good summary to me. I think going from 3 to 4 has been the step I've always been pretty fuzzy on (though I'm guessing you know this pretty well 😉). How does one turn a Conda package tarball into something that pyodide can use? Where do these get hosted?

As Isuru's point about adding a platform in Conda, I think this was the idea behind issue ( conda/conda#7619 ), but may be missing things here.

One other thing worth discussing is over time LLVM's ability to build WASM binaries has grown. Does it make sense to start using that or is Emscripten still needed for some things? The one thing that I recall use to be a stumbling block was a libc implementation for WASM. Though it looks like this has since been solved? In which case maybe building this libc binary gives us enough to get started?

@isuruf

Thanks for the feedback @isuruf @jakirkham ! Great to know that cross compilation works conda-forge.

I think going from 3 to 4 has been the step I've always been pretty fuzzy on

As far as I understand those (.data, .js) are files to unpack in the virtual filesystem and associated unpacking instructions (cf emscripten docs). We could potentially repackage them into a single file during build then separate them on the client side.

The question for me is more is the conda artifacts really the best format for distributing such files (which were not optimized for this use case). Maybe there are some other approaches in the JS/WASM ecosystem that would be better. I haven't studied the question in detail so far.

At present, we put the artifacts in an S3 under versioned paths and CDN proxy it by JsDelivr, which allows for a lot of flexibility.

One other thing worth discussing is over time LLVM's ability to build WASM binaries has grown. Does it make sense to start using that or is Emscripten still needed for some things?

Absolutely, it would be good to revisit if the situation evolved much since the discussion in conda/conda#7619

The one thing that I recall use to be a stumbling block was a libc implementation for WASM. Though it looks like this has since been solved?

The link you provide is for WASI or outside of the browser, as far as I understand though? Within the browser, we still need an in-memory filesystem (currently provided by emscripten) as CPython wouldn't be very useful without it. For instance Rust can be easily compiled to WASM (without emscripten), but then you cannot do filesystem I/O at least as indicated in the reference materials. Maybe there are indeed lighter projects providing the necessary abstractions, it could be worth checking.

I have just updated binaryen (conda-forge/binaryen-feedstock#38) and added an Emscripten recipe to conda-forge (conda-forge/staged-recipes#13178). With those two things we could be ready to explore these ideas.

I'd be happy to integrate cross-compilation to javascript "natively" into boa so that it becomes very straight-forward ... although I am not sure how simple that will be.

I think conda is more than the package format, and that should not be the thing holding us up -- we can easily create a new, more web-optimized package format (different or no compression etc.) if we have to.
What I think is great is the community, the available packages, automation, and the build infrastructure by conda-forge etc.

Regarding libc, I was under the impression that emscripten did some magic to get me a libc :) in general, emscripten seemed to work quite smoothly.

Personally, I'd be quite interested in getting the packages in boa-forge to compile to see if we can bootstrap a wasm-micromamba.

I have just updated binaryen (conda-forge/binaryen-feedstock#38) and added an Emscripten recipe to conda-forge

If you don't use emsdk you might need to package it manually which doesn't sound that simple (but they did it in Homebrew)

@rth

It's not very difficult to package emscripten - I have a working recipe that I have been using to build wasm conda packages.
It's surprisingly simple, and I already got working libraries for a bunch of compression algorithms that are of interest to us, as well as the first half of libsolv (I managed to run it to parse conda repodata.json into a solv file.

So it all sounds quite doable (including running a micromamba in the browser). For that we'd need to rewrite mamba a bit though, to use the emscripten Fetch and FileSystem APIs instead of curl. We could give it a shot though.
Alternatively we could figure out how make a mamba-derivative that one can feed repodata.json through javascript and that spits out the packages to download, and then unpacks that using a different entrypoint or something like that...

@rth you might know better how to do this!

We could also make sure to only support one of the two (three, actually) compression algorithms that conda is using currently (e.g. allow only .tar.bz2 packages?!).
I have a small issue compiling zstd, it tells me that wasm-ld: error: initial memory too small, 38821440 bytes needed (any ideas, @rth ?)

I can upload the recipes I have soon. Would be cool to have a collaboration for this!

@wolfv

Sorry for slow response @wolfv

It's not very difficult to package emscripten - I have a working recipe that I have been using to build wasm conda packages.

That's great!

For that we'd need to rewrite mamba a bit though, to use the emscripten Fetch and FileSystem APIs instead of curl.

Indeed. In pyodide, we just mount the filesystem and then it can be interacted with directly from Python. However for interacting with remote URLs one indeed need to rewrite everything with Web APIs (e.g. pyodide.open_url)

I have a small issue compiling zstd, it tells me that wasm-ld: error: initial memory too small, 38821440 bytes needed

Likely https://github.com/emscripten-core/emscripten/blob/1216d230eac6a335f1397f4ab1d2bf297113633b/src/settings.js#L152 needs to be increased with a env variable.

Would be cool to have a collaboration for this!

Yes, that would be great. For pyodide that would mean incrementally get closer to the conda-forge apporach and possibly start using some of the tooling.

One thing where I would be interested in your feedback is assuming python packages are built with emscripten on conda-forge, would the current way of detecting it at build time in setup.py is appropriate (currently via the PYODIDE_PACKAGE_ABI env variable) or would you propose some other solution? It would be good to agree on that before that variable end up in too many upstream projects. Or do you think that just a env variable for cross-compilation (e.g. as done by @isuruf in scikit-learn/scikit-learn#18884) would be enough?

Are you interested in helping me to bootstrap some "emscripten" enabled recipes? I can create a repo, and we could run a Azure pipeline to build a couple of wasm compiled conda packages.

@wolfv

@wolfv yes, we could try to experiment there.

Ok, I'll have to get emscripten on conda-forge first, then I'll set it up.

API Rename packages.json to repodata.json (#2749)

So overall emscripten-forge went this road, we are more focused on producing wheels with PyPA / cibuildwheel tooling. But we can certainly open more specific issues about ways to share some of the tools or approaches.

@DerThorsten

Should they mention MambaLite

Happy to add emscripten-forge under related projects (please open a PR) however I don't think this belongs, at least for now, in the main section on how to install packages (which is already rather confusing ). They are alternative distributions. For instance, you won't see in the pip documentation, "well you can also install this with conda", or conda-forge advertising to use homebrew in the official documentation.

Though in any case, it would be good to figure out binary compatibility first between Pyodide and emscripten-forge packages

Also, I'm very happy to chat @DerThorsten and see points on which we could work together. For instance, we are currently unvendoring micropip from the monorepo so it's easier to reuse if necessary #3093

My point in closing this issue was that we can open more specific discussion points, but we can also continue the discussion here if you prefer.

@westurner thank you for your comments with this information, but I'm not entirely sure what you proposing though :)

@DerThorsten

Though in any case, it would be good to figure out binary compatibility first between Pyodide and emscripten-forge packages

agreed, that would be a great first step!

Also, I'm very happy to chat @DerThorsten and see points on which we could work together. For instance, we are currently unvendoring micropip from the monorepo so it's easier to reuse if necessary #3093

Would be happy to work together more!

@westurner I have no clue what you are proposing

This issue having been closed, it seemed out of the way from actual progress that it might be holding up.

There are many systems for package metadata and signed cryptographic manifests (some with per-file hash checksums). Where we have functional overlap and duplication of effort, there is potential for security vulnerability.

How should pyodide's build change to better integrate with conda-forge (and emscripten-forge)? Hopefully the aforementioned tools copy the manifest signatures over when re-packing and re-hosting.

When pyodide / micropip was written:

pip didn't have end-to-end-package signatures (~~GPG w/o keyserver~~)
conda didn't have end-to-end package signing
GPG asc signature upload support may have already been removed from warehouse (the application that hosts PyPI)
TUF server signatures were not yet implemented on PyPI
(TUF author signatures are not yet implemented on PyPI (end-to-end package signing))
Signed HTTP Exchanges (SXG) & Web Bundles weren't yet finalized (could be used to secure WASM packages)

HTTP Subresource Integrity (SRI):

<script src="https://cdn.example.com/app.js"
       integrity="sha384-+/M6kredJcxdsqkczBUjMLvqyHb1K/JThDXWsBVxMEeZHEaMKEOEct339VItX1zB"
       crossorigin="anonymous"></script>

There are many opportunities to drop the ball in build systems and application dependency composition; DevSecOps for software supply chain security.

How can {pyodide, micropip, mambalite,} require package signatures (to a standard better than the tools they build atop) in order to prevent (widescale) exploitation of browsers with WASM and no quotas and someday, local File System Access?

Yes, these are good questions. Would you mind though opening a separate issue about end-to-end package signing, as it's a very specific technical point that it would be better to discuss separately from this very general issue?

In our case, as compared to pip or conda, the attack surface is very much reduced due to the browser sandbox. But yes, the security story can certainly be improved and there is still the usage in Node which doesn't provide a sandbox.

How should pyodide's build change to better integrate with conda-forge (and emscripten-forge)?

Short answer is we don't plan to integrate with conda forge anymore. emscripten-forge is working on that. For Pyodide we are going for better integration with PyPI, PyPA tooling, cibuildwheel etc.

Are there projections as to what the load impact on PyPI and it's CDN expenses will be from WASM apps pulling dependencies on every run? https://pypi.org/sponsors/ - "[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 - Pip downloads wheels for every CI build and deployment - Pip does not download wheels for every process invocation - Micropip downloads wheels for every page load process invocation from PyPI - MambaLite downloads empkg rebuilds of conda packages unless there's a noarch conda package, but their HTTP headers aren't setup to CDN for "download every package on every process invocation / page load"

…

On Sun, Sep 18, 2022, 12:20 PM Roman Yurchak ***@***.***> wrote: Yes, these are good questions. Would you mind though opening a separate issue about end-to-end package signing, as it's a very specific technical point that it would be better to discuss separately from this very general issue? In our case, as compared to pip or conda, the attack surface is very much reduced due to the browser sandbox. But yes, the security story can certainly be improved and there is still the usage in Node which doesn't provide a sandbox. How should pyodide's build change to better integrate with conda-forge (and emscripten-forge)? Short answer is we don't plan to integrate with conda forge anymore. emscripten-forge is working on that. For Pyodide we are going for better integration with PyPI, PyPA tooling, cibuildwheel etc. — Reply to this email directly, view it on GitHub <#795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMNS2LCDVBMQ5A57EMALTV646OBANCNFSM4TP353FA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

(I must have confused "building pyodide" / "building pyodide packages like conda-forge / emscripten-forge" with "just have piplite solve and install the dependency graph from PyPI for every page load")

…

On Sun, Sep 18, 2022, 2:30 PM Wes Turner ***@***.***> wrote: Are there projections as to what the load impact on PyPI and it's CDN expenses will be from WASM apps pulling dependencies on every run? https://pypi.org/sponsors/ - "[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 - Pip downloads wheels for every CI build and deployment - Pip does not download wheels for every process invocation - Micropip downloads wheels for every page load process invocation from PyPI - MambaLite downloads empkg rebuilds of conda packages unless there's a noarch conda package, but their HTTP headers aren't setup to CDN for "download every package on every process invocation / page load" On Sun, Sep 18, 2022, 12:20 PM Roman Yurchak ***@***.***> wrote: > Yes, these are good questions. Would you mind though opening a separate > issue about end-to-end package signing, as it's a very specific technical > point that it would be better to discuss separately from this very general > issue? > > In our case, as compared to pip or conda, the attack surface is very much > reduced due to the browser sandbox. But yes, the security story can > certainly be improved and there is still the usage in Node which doesn't > provide a sandbox. > > How should pyodide's build change to better integrate with conda-forge > (and emscripten-forge)? > > Short answer is we don't plan to integrate with conda forge anymore. > emscripten-forge is working on that. For Pyodide we are going for better > integration with PyPI, PyPA tooling, cibuildwheel etc. > > — > Reply to this email directly, view it on GitHub > <#795 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAMNS2LCDVBMQ5A57EMALTV646OBANCNFSM4TP353FA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

@westurner

@westurner I opened #3127 for CDN resource usage discussion.

Just as an FYI, I opened conda-forge/staged-recipes#20961 adding micropip as a conda-forge noarch package, which should (I hope? idk :D) allow emscriptenforge and micropip to be used together like how conda-forge and pip can be used together.

@yuvipanda

awesome @yuvipanda – that's great news. It didn't even come to my mind to just add it to conda-forge :)

@yuvipanda

@yuvipanda

ENH: A micropip.main() for %pip in Jupyter micropip#82

@rth

Hi @rth I was curious to know what was the status of supporting conda-forge packages with pyodide! :-)

jupyterlite-xeus: https://github.com/jupyterlite/xeus :

The core feature of jupyterlite-xeus is its integration with

emscripten-forge, a conda package distribution tailored for WebAssembly. This makes it possible to bundle your favorite scientific or data analysis packages directly into the browser-based environment, delivering a reproducible computing experience with zero backend dependencies. Docs: https://jupyterlite-xeus.readthedocs.io/en/stable/#usage :

Once installed, you can create an environment.yml file at the root of

your jupyterlite build directory containing the following: ```yaml name: xeus-kernels channels: - https://prefix.dev/emscripten-forge-4x - https://prefix.dev/conda-forge dependencies: - xeus-python - xeus-lua - xeus-nelson - numpy - matplotlib - pillow - ipywidgets - pip: - ipycanvas ```

Starting with jupyterlite-xeus v4.0.0a11, you can use the %pip and %mamba

magics or the !pip and !mamba commands to install packages dynamically .. https://github.com/jupyterlite/repo2jupyterlite https://github.com/jupyterlite/xeus-lite-demo :

Xeus-Lite demo This GitHub template allows you to create deployments of JupyterLite with

a custom set of conda packages Re: repo2jupyterlite and container2wasm c2w: container2wasm/container2wasm#550 (comment)

…

On Wed, Apr 15, 2026, 3:06 PM Rémi Delaporte-Mathurin < ***@***.***> wrote: *RemDelaporteMathurin* left a comment (pyodide/pyodide#795) <#795 (comment)> Hi @rth <https://github.com/rth> I was curious to know what was the status of supporting conda-forge packages with pyodide! :-) — Reply to this email directly, view it on GitHub <#795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMNSYQ5XJ7RF54CVQHJND4V7MUPAVCNFSM6AAAAACX2TTB3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DENJUG4YTMMRRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

I would say we are not interested in supportding conda-forge anymore. As PEP 783 is accepted now, we are mainly focusing on publishing packages to PyPI, which should be consumable by emscripten-forge or jupyterlite people as well.

Navigation Menu

Repository navigation

Better integration with conda/conda-forge for building packages #795

1. Updating emscripten

2. Build approach

3. Shared package specifications

4. Artifacts format

5. Dependency resolution in the browser

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Better integration with conda/conda-forge for building packages #795

Description

1. Updating emscripten

2. Build approach

3. Shared package specifications

4. Artifacts format

5. Dependency resolution in the browser

Activity

isuruf commented on Nov 9, 2020

jakirkham commented on Nov 9, 2020

rth commented on Nov 9, 2020

wolfv commented on Nov 9, 2020

rth commented on Nov 10, 2020

wolfv commented on Nov 21, 2020

rth commented on Nov 30, 2020

wolfv commented on Dec 1, 2020

rth commented on Dec 1, 2020

wolfv commented on Dec 1, 2020

rth commented on Sep 17, 2022

4 remaining items

rth commented on Sep 18, 2022

DerThorsten commented on Sep 18, 2022

westurner commented on Sep 18, 2022

rth commented on Sep 18, 2022

westurner commented on Sep 18, 2022

westurner commented on Sep 18, 2022

hoodmane commented on Sep 18, 2022

yuvipanda commented on Nov 1, 2022

wolfv commented on Nov 1, 2022

westurner commented on Sep 11, 2023

RemDelaporteMathurin commented on Apr 15, 2026

westurner commented on Apr 17, 2026

ryanking13 commented on Apr 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions