Boom! Michael Droettboom's blog - mozilla

Profiling WebAssembly

2018-04-11T00:00:00-04:00

Summary: Tips for profiling WebAssembly

I couldn't find a comprehensive guide to profiling WebAssembly, so I thought I'd share my own limited experience here. In my last post, I talked about benchmarking a WebAssembly port of the scientific Python stack. I knew which benchmarks were doing better than others and had some theories about why, but since I didn't yet know how to profile WebAssembly, I couldn't really answer that with any certainty.

It turns out that profiling WebAssembly is quite easy.

Rebuilding with the --profiling flag

The first step is to rebuild the application with the --profiling flag passed to both the compiler and the linker for every object. This makes sure that all of the information necessary for profiling is available in the output and makes the code more readable. The typical way to do this would be to set CFLAGS and LDFLAGS and let the ./configure script for your project pick those up. In the case of pyodide, the Python cross-compiling setup makes that tricky, or at least I couldn't figure it out in a short amount of time. Fortunately, emscripten provides a handy backdoor to just force this on everything: the EMCC_CFLAGS environment variable. Therefore, to make a profiling-friendly build of pyodide:

make clean
EMCC_CFLAGS=--profiling make

Setting start and stop points for profiling

You generally don't want to profile an entire run, which would include initialization and other things. It turns out there's a handy Javascript API to turn the profiler on and off.

console.profile() turns the profiler on.
console.profileEnd() turns the profiler off.

If you wanted to call these from C/C++, you could use the EM_ASM macro, which allows you to insert literal Javascript into the C application:

EM_ASM(
    console.profile();
);

In my case, I wanted to turn the profiler on and off from Python, so I can do:

from js import console
console.profile()

Profiling

The actual profiling is performed within the development tools of your browser. When you load an .html file that runs the WebAssembly built and instrumented as above, it will record a set of profiling data available from the Performance tab.

I'll refer you to the Performance Tools documentation for more information. Suffice it to say that profiling WebAssembly is almost exactly like profiling vanilla Javascript in the browser.

Case study

For pyodide, I created a profiling build to look into the julia benchmark that I knew was performing poorly. Right away, I noticed from the Call Tree that 50% of the time was spent in this function:

for (var named in NAMED_GLOBALS) {
    (function(named) {
        Module['g$_' + named] = function() {
            return Module['_' + named] // <- 50% of runtime HERE
        };
    })(named);
}

This code is actually part of the boilerplate that emscripten emits. It helps dynamically loaded modules (such as Numpy in my case) access symbols in the main module. Since these symbols don't change at runtime, we don't actually need to do the dictionary lookup for Module['_' + named] every time, we can cache (memoize) it at startup and then just use that:

for (var named in NAMED_GLOBALS) {
    (function(named) {
        var func = Module['_' + named];
        Module['g$_' + named] = function() {
            return func;
        };
    })(named);
}

This 2-line change to emscripten resulted in significant speedups in my pyodide benchmarks across the board.

description

Here, the x-axis is the number of times slower that WebAssembly runs vs. native code. The grey bars are the timings before this change, and blue bars are the timings after this change.

More details about this changes are in the pull request.

Scientific Python in the Browser

2018-04-04T00:00:00-04:00

Summary: An early report on getting the scientific Python stack compiled to WebAssembly.

Data Science in the Browser

Shortly after starting at Mozilla in January, I became aware of Hamilton Ulmer and Brendan Colloran's Iodide project, an experiment to build a data science notebook based on web technologies. Unlike Jupyter notebooks, the computation happens in the browser, with direct access to Web API technologies like the DOM. Sharing a notebook is as simple as passing around a single HTML file, since there's no server side to worry about. It's not out to replace Jupyter notebooks, but rather to exist in a different design tradeoff space that makes it more suitable the sharing and collaboration.

Since it targets the browser, the programming language of Iodide is, of course, Javascript. While there are a number of libraries for doing data science in Javascript, such as numjs and scijs, they aren't as widely used or as battle-tested as the scientific Python or R ecosystems. Nonetheless, I think "data science in Javascript" is an interesting area to explore, particularly since Javascript has some of the best JIT compilers of any dynamic language. This advantage allows writing both high-level orchestration and low-level numeric code in the same language, side-stepping the notorious "two language problem" in scientific Python. (In Python land, most of the core scientific libraries have significant chunks of code in lower level languages such as C, FORTRAN or Cython for performance reasons.) Combining Javascript's great compiler technology, and perhaps adding a smattering of transpilation to fix some syntactic issues, is really promising, and Iodide as a project is exploring that space.

Nonetheless, we received frequent feedback that Iodide "looks really cool, but I wish I could use the Python (or R) tools I'm familiar with." I understood in theory that it should be possible to compile the Python interpreter into WebAssembly in order to run it in the browser. There are already a few projects that do this: (cpython-emscripten, micropython javascript support, pypyjs). Unfortunately, I couldn't find a project that included a practical scientific Python stack including Numpy and friends. I was concerned about the amount of effort it would take to build such a thing, and also whether the result would be performant enough to be useful. In February, we had a conversation with some folks who work on WebAssembly tooling at Mozilla, and they were pretty bullish that it wouldn't be too hard. Based on their optimism, I gave it a shot, and starting with dgym's cpython-emscripten as a basis, I had the basic parts of a Python interpreter working in WebAssembly in a couple of days. Of course, going from that to a working Numpy took much longer, but thanks to some help from Alon Zakai and others, Numpy is working, too. With that done, it has been much easier getting other libraries higher up the stack to work, including preliminary support for Pandas.

Tight integration

One thing that sets this implementation apart from other Python-in-the-browser projects I've come across is the ability to easily pass and share objects between Python and Javascript.

The basic Python data types (None, bool, int, float, str, bytes, list and dict) are transparently converted to and from their Javascript equivalents. Other types, including Numpy arrays, are wrapped in a proxy that allows Javascript to call their methods and access their items and attributes. Vice versa, Javascript objects are wrapped in a Python proxy. These proxies allow objects to be shared on both sides of the language barrier without copying, which is particularly important for large Numpy arrays.

Say, for example, you had a value in Javascript:

// javascript
secret = "Wklv#lv#olnh#pdjlf$"

You could use it from Python by using the from js import ... syntax:

# python
from js import secret

decoded = ''.join(chr(ord(x) - 3) for x in secret)

And then send data back to the Javascript side using pyodide.pyimport:

// javascript
var decoded = pyodide.pyimport("decoded")

One of the coolest side effects of this design is that Python has complete access to the Web API, so it can manipulate the DOM, use HTML Canvas, access webcams or audio and all the other cool things you can do from Javascript in a browser.

For example, changing the browser tab's title is as simple as importing window and setting an attribute:

from js import window
window.title = "My mind is blown"

What works

Most of the Python standard library works. The most notable exceptions are:

subprocess: since the browser isn't an OS, it can't spawn new processes.
socket: access to raw network sockets would break the browser security model. There are a lot of networking-related things in the standard library built on socket that therefore also don't work.
All of the browser sandboxing still applies, so you can't access the local filesystem. However, by calling through Javascript, you do have access to XMLHttpRequest and browser local storage. Eventually, Python wrappers around this functionality should be written to make those operations feel more like they do in native Python.

Within Numpy, all of the core functionality works, but there's no support for long double (but those are pretty niche). There are still some low-level compiler bugs that prevent the FFT stuff from compiling, but that should eventually resolve.

How fast is it?

To answer this question, I reached for a few existing Python and Numpy benchmarks:

The venerable pystone, which ships with CPython.
Serge Guelton's set of numpy benchmarks.

These benchmarks probably fall into the trap of being a little too "synthetic". I would have preferred to also use the Python Performance Benchmark Suite, which aims to be a little closer to "real world", but it has a significant number of dependencies and would need to be adapted to work on a platform without subprocess before it could be used in this context. Nonetheless, I think these benchmarks offer a useful approximation for now.

The benchmarks were run on the same machine in the native CPython implementation and in Firefox Nightly using selenium. The following figure shows how many times slower the WebAssembly implementation is.

description

EDIT 2018-04-10: The original results posted here inadvertently included Numpy import time in the WebAssembly times (but not in the native times). These have now been corrected above. There is some improvement in the results, but not in a best or worst case. You can see the original results here.

The results are interesting. For benchmarks that spend most of their time in Numpy routines, such as harris or rosen, runtime is at par with the native-compiled Python. When WebAssembly rocks, it really, really rocks. Unfortunately, for other benchmarks that spend a lot of time looping or making function calls in Python, runtimes can be as much as 35 times slower. I have an unsubstantiated hunch that this is due to the use of Emscripten's EMULATE_FUNCTION_POINTER_CASTS option which is required to make all of the function pointer calls that CPython does work correctly.

UPDATE 2018-04-11: My hunch was wrong, and I was able to get to the bottom of the root cause and significantly speed up these benchmarks. See my post Profiling WebAssembly for more info.

Future directions

I'd love to see improvements to the toolchain that close the performance gap. At this point, I don't personally know enough to anticipate how much work is involved.

Another current limitation is that all of the packages you anticipate you might need must be compiled and wrapped into a single large data file that is downloaded in its entirety to your browser before anything can start. It would be great to modularize that, so that packages are downloaded on demand. Related to that, it would also be helpful to modularize the build system so that individual packages can be added more independently. Conda build could potentially serve as a basis for that.

Check it out

The easiest way to play with this is to visit the example Pyodide notebook (EDIT: This link was fixed to a working version). (Note that this only works on Firefox right now. Chrome support is pending).

You can also get involved at pyodide github repository. Note that while Pyodide grew out of the needs of Iodide, there's nothing Iodide-specific about it, and it should be useful in other contexts where you want to embed a scientific Python stack in the browser. I'm pretty new to WebAssembly and I'd love any help, advice or comments to make this better.

Jupyter notebooks and version control

2018-01-18T00:00:00-05:00

Summary: Presents an experimental alternative file format for Jupyter notebooks that plays nicer with version control.

UPDATE: 2018-04-11: Matthias Bussonnier pointed out on Twitter he did something very similar quite some time ago. Given the interest, what can we do to push this along so it's an obvious choice for Jupyter users?

The problem

There's no doubt that Jupyter notebooks are taking over the world for data science experimentation. When notebooks are relied on for ongoing decision-making within an organization, it's inevitable that, like all software, they will require bugfixes or updates. Without proper version control of these changes, its difficult to know what changes were made, and, more importantly, to reason about what effect those changes may have had on the results. While you can simply put Jupyter notebooks into a version control system (VCS), such as git, the design of the notebook file format makes certain important operations, like calculating the difference between two revisions, less friendly than they could be.

For example, the Jupyter notebook file format contains binary blobs for image output. The diffs between these sections are large and noisy, and ultimately unhelpful to the software developer reviewing a pull request. Secondly, since the file format is based on JSON, multi-line strings (such as all source code) are full of boilerplate: Each line is in its own set of double-quotes, with explicit newlines (\n). For example, this Python:

items = []
for i, item in enumerate(database.all_docs(params={'include_docs' : True})):
    if i > 1: break
    items.append(item)
print items

turns into the following JSON:

"source": [
  "items = []\n",
  "for i, item in enumerate(database.all_docs(params={'include_docs' : True})):\n",
  "    if i > 1: break\n",
  "    items.append(item)\n",
  "print items"
]

All of this makes it more difficult to see meaningful changes through all the noise.

Some solutions

I'm by no means the first person to notice these issues, and others have tackled this problem from different directions.

In the blog post Making Git and Jupyter play nice, Tim Staley suggests filtering the notebook files to remove the output cells and less important content (such as metadata or execution_count). While this goes a long way to removing a lot of the "noise" in diffs, the content is still JSON.

ipymd perhaps comes the closest to solving this problem, in my opinion, by converting notebooks to markdown, with code inserted as standard markdown code blocks. For example:

Here is some Python code:

```python
>>> print("Hello world!")
Hello world!
```

It currently has a major shortcoming, in that it doesn't handle non-textual output cells. Jupyter output cells are more complex than they might appear at first glance, as each can include multiple representations of the same thing, allowing the front-end to ultimately choose the best based on context. I think there are probably some clever ways that could be resolved, but any solution is likely to be lossy relative to what standard Jupyter notebooks can do, or break strict compatibility with markdown.

There is also nbdime: a tool designed specifically for diffing and merging Jupyter notebooks. The advantage of nbdime is that, since it was purpose-built for Jupyter notebooks, the user interface can take advantage of notebook-specific features, such as image diffing. Unfortunately, it is hard to integrate it into existing code review workflows, like Github pull requests. Here at Mozilla, for example, many of those doing data science in Jupyter notebooks are Firefox software engineers first and foremost, so they are comfortable and opinionated about the community of "power tools" built around version control: suggesting a single-use tool just for notebooks might be a hard sell.

Yet another idea

Given all this, I've experimented with an approach that both allows for better diffs while still retaining all of the information present in a Jupyter notebook. There are lot of ways it could be done, but in the interest of not inventing entirely new syntax, it turns out you can get a large part of the way to the goal by just using YAML instead of JSON, with the following tweaks to the YAML writer:

Always write markdown, source code and text output in literal style. This avoids the explicit newlines required for multi-line text in JSON.
Remove any keys that point to empty lists or dictionaries, e.g. a metadata entries without any actual metadata.
Order the keys in a consistent way that makes sense to humans. For example, for each cell include the cell_type first, followed by the content, with less important things like metadata and execution_order afterward.
Non-textual output cells are saved externally into a directory of files. This keeps output out of the diff of the main code in the notebook, but it's still present when the notebook is reloaded.

For example, here is a cell with output in this YAML-based format:

- cell_type: code
  source: |-
    fig, ax = plt.subplots(figsize=(18,10))
    sns.boxplot(df.ARR_DELAY_NEW, df.FL_DATE, ax=ax)
    fig.autofmt_xdate()
  outputs:
  - output_type: display_data
    data:
      text/plain: |-
        <matplotlib.figure.Figure at 0x7f679ad6e190>
      image/png: Exploration of Airline On-Time Performance_files/6ff8d173e4d8a288.png
  metadata:
    collapsed: false
  execution_count: 45

Compare that to the original in standard Jupyter notebook JSON:

{
 "cell_type": "code",
 "input": [
  "fig, ax = plt.subplots(figsize=(18,10))\n",
  "sns.boxplot(df.ARR_DELAY_NEW, df.FL_DATE, ax=ax)\n",
  "fig.autofmt_xdate()"
 ],
 "metadata": {
  "collapsed": false
 },
 "outputs": [
  {
   "metadata": {},
   "output_type": "display_data",
   "data": {
    "text/plain": [
     "<matplotlib.figure.Figure at 0x7f679ad6e190>"
    ],
    "image/png": "...BASE64 encoded data removed...",
   }
  }
 ],
 "execution_count": 45
},

The result is something that is not quite as user-friendly as the markdown produced by ipymd, but it is fully lossless. Another nice feature of the design is that converting from this format back to a standard Jupyter notebook is as simple as loading YAML, snarfing the external content back in place, and writing out JSON. That hopefully bodes well for its future-proofing as Jupyter continues to evolve.

One problem still present is that markdown cells don't diff very well, since in most cases markdown paragraphs are written as one long continuous line. Brandon Rhodes has some great suggestions about using semantic linefeeds to make prose more easily diffable that would help there, but I don't think that is 100% automatable.

Playing with the idea

I have an experimental plugin for nbconvert on Github that implements the conversion to and from this YAML-based format. As an experiment, I ran the conversion over the entire git history of a collection of Jupyter notebooks put out by IBM Emerging Technologies. (git filter-branch is an awesome tool for this exact purpose, by the way.) Many of the notebooks in this repository have no history (whether that's because versioning Jupyter notebooks is too hard, we may never know), but for those that do have history, there is definitely some useful improvement, for example compare before and after.

My main purpose of this blog post is just to solicit feedback on these ideas as I work toward a solution for better support for version control workflows with Jupyter notebooks. Please leave comments, suggestions and questions below.

Acknowledgements

This work was supported by moz://a.