• Introduction
• User-Defined Types
• A Word on Typed Arrays
• Honorable mention: v8::ValueSerializer
• @ungap/structured-clone
• @worker-tools/structured-json (Typeson)
• MessagePack + Extensions
• CBOR + Standard Extensions
• Conclusion

Introduction

The Structured Clone Algorithm (SCA) is the web’s canonical way of serializing/deserializing JavaScript objects. It is used when passing objects via postMessage calls to workers or iframes, or when storing objects in IndexedDB. One major benefit over JSON is support for ECMAScript types like Map, Set, Date, RegExp and Error, as well as typed arrays such as Uint8Array.

It also supports various platform objects, such as File, Blob, and FileList, as well as more unusual ones, like DOMMatrix or VideoFrame. For the purpose of this article we will only check for support for Blob and File, which have implementations outside the browser.

In addition to the ECMAScript types and platform objects mentioned above, the SCA also supports preserving object identity and circular references. If the same object is present in the source multiple times, the identities in the cloned object will reflect the original identities. This is a big advantage over JSON and something that is frequently requested. For example, the flatted module on npm has over 29 million downloads per week.

const obj = { a: 3 };
const clone = structuredClone({ main: obj, copy: obj });
clone.main === clone.copy // true

While all JS engines implement the Structured Clone Algorithm, and even expose it trough the structuredClone builtin function, they do not support serialization to a buffer or provide access to its internal binary representation (which isn’t standardized), leaving it in developers hands how they want to achieve similar results.

User-Defined Types

When the topic of serialization comes up, usually it is implied that there is also a way to serialize user-defined types. This is not what the SCA is about. The SCA is closer to “Extended JSON” than Protocol Buffers and as such provides no mechanism for developers to add custom types.

When the algorithm encounters an unknown class instance, it will extract its enumerable properties and put them in a POJO, with no corresponding step on the deserialization side. Interestingly, it will even ignore the toJSON method (it is not JSON after all).

Perhaps this is for the better, since executing arbitrary constructors based on wire data can be a security issue. Of course, such a mechanism can always be layered on top through a pre- and post-processing step, at the cost of performance and inconvenience. For this reason, it will be noted on each candidate if there is a way to define custom “transferrable” types.

A Word on Typed Arrays

Typed Arrays deserve special attention in this context. It is perhaps poorly understood that JS’ typed arrays are actually “Array Buffer Views”, as in, they provide a view on an underlying ArrayBuffer. In the most common scenarios this distinction is irrelevant, such as when creating a fresh array:

crypto.getRandomValues(new Uint8Array(16))

…or receiving one from a platform method:

new TextEncoder().encode("Hello World")

In these cases, a new ArrayBuffer of the exact size of the Uint8Array is created and the developer needn’t worry about it. However, this isn’t the only way typed arrays can be used. Take the following example:

const ab = new ArrayBuffer(1024);
const view8 = new Uint8Array(ab, 0, 8);
const view16 = new Uint16Array(ab, 0, 4);

Here, both views represent a view on the first 8 bytes of the same underlying array buffer. Modifying them in one view, will reflect on the other and vice versa. More importantly, strictly following the SCA when serializing these views means preserving the object identity of the buffer, meaning all 1024 bytes have to be included in the serialized output, even though we’re most likely only interested in the first 8 bytes!

Not making a distinction between a view and the underlying buffer can be a source of bugs. SQLite Viewer for VSCode had two bugs because I didn’t account for the fact that an Uint8Array can have a larger underlying array buffer. In my case, a extra byte was prepended, which made the SQLite format unreadable.

Every serialization implementation can decide whether it wants to preserve the object identity of the underlying buffer property and include it wholesale, or if it wants to treat typed arrays like slices and only inline its particular contents. The web platform is clear on which version it prefers:

const ab = new ArrayBuffer(4096)
const view1 = new Uint8Array(ab, 0, 8);
const view2 = new Uint8Array(ab, 8, 16);
const [view1Copy, view2Copy] = structuredClone([view1, view2]);
view1Copy.buffer === view2Copy.buffer // true
view1Copy.buffer.byteLength // 4096

However, for the purpose of storage or sending messages across a network, inlining just the slice that a view provides is probably closer to what developers would expect, even if the underlying buffer identities are not preserved in that case.

Honorable mention: v8::ValueSerializer

V8 has an internal serialization format that implements the SCA, is backwards compatible (can be stored), and is the closest thing to a canonical binary representation for JS objects. While it’s unlikely to be shared by other JS engines, given V8’s prevalence, it is a strong contender for a default format.

Unfortunately, it is poorly documented and has no pure JS implementation that make it useable outside of the V8 contexts in which it is exposed it to developers (i.e. node:v8 module in Node and Deno). It also has no libraries for other languages, making it a poor choice for an exchange format.

UPDATE: Since writing this, I took a deeper look into the V8 serialization format. There’s a high degree of attention to detail when it comes to JavaScript’s peculiarities, such as primitive wrappers (e.g. new Number(3)), holes in arrays and a number of other edge cases. None of the other formats discussed here have the same level of compatibility with the SCA. The format is also neat in other ways and it so happens that I’ve ported it from C++ to TypeScript, which you can check out here: https://jsr.io/@workers/v8-value-serializer

@ungap/structured-clone

@ungap/structured-clone is a pure JS library that implements the SCA without any opinions on binary serialization.

For example the serialization for an object like this:

const ab = new ArrayBuffer(8);
const data = new Uint8Array(ab).fill(255);
const view = new Uint16Array(ab);
const obj = {
  big: 2n**64n - 1n,
  set: new Set([1,2,3]),
  exp: new RegExp("/^\d+$/", "g"),
  map: new Map([["a", 3]]),
  data,
  view,
}
obj.self = obj; // add circular refernce

will produce a JS array that looks like this:

[
  [ 2, [ [ 1, 2 ], [ 3, 4 ], [ 8, 9 ], [ 10, 11 ], [ 13, 14 ], [ 15, 16 ], [ 17, 0 ] ] ],
  [ 0, "big" ],
  [ 8, "18446744073709551615" ],
  [ 0, "set" ],
  [ 6, [ 5, 6, 7 ] ],
  [ 0, 1 ],
  [ 0, 2 ],
  [ 0, 3 ],
  [ 0, "exp" ],
  [ 4, { source: "\\/^d+$\\/", flags: "g" } ],
  [ 0, "map" ],
  [ 5, [ [ 12, 7 ] ] ],
  [ 0, "a" ],
  [ 0, "data" ],
  [ "Uint8Array", [ 255, 255, 255, 255, 255, 255, 255, 255 ] ],
  [ 0, "view" ],
  [ "Uint16Array", [ 65535, 65535, 65535, 65535 ] ],
  [ 0, "self" ]
]

import * as ungap from 'npm:@ungap/structured-clone';
ungap.deserialize(ungap.serialize(obj));

While this lends itself well for further JSON or binary serialization, doing so has a number of disadvantages:

• It is barely human-readable, comparable to a fully fledged binary format
• When used in combination with JSON, the overhead of inlined binary is larger than even the 33% penalty of using Base64 encoding.
• It doesn’t inline buffers well when using binary encodings like MessagePack either, because each byte is a number in a regular array. This means every number above 127 will be prefixed with an extra header byte¹, leading to a ~50% penalty².
• It does not support common platform objects like Blob and File.
• It has no hooks for handling custom types
• No support for BigUint64Array and BigInt64Array

Deserializing the object will preserve circular references, but not the identity of the underlying array buffer. As mentioned earlier, this is a reasonable choice and probably even preferable if accuracy isn’t the primary goal.

On the flip side, the code is simple and compact and has no external dependencies.

@worker-tools/structured-json (Typeson)

This is my own entry to the mix, though it is really just a thin wrapper around the Typeson library’s structured clone preset, which does the heavy lifting.

Like the ungap version, it is intended to produce JSON output, but instead of a custom array-based format, it attaches a $types property that carries meta information.

import * as SJSON from 'npm:@worker-tools/structured-json';
SJSON.revive(SJSON.encapsulate(obj));

Using the same obj as above, it will produce the following POJO:

{
  big: "18446744073709551615",
  set: [ 1, 2, 3 ],
  exp: { source: "\\/^d+$\\/", flags: "g" },
  map: [ [ "a", 3 ] ],
  data: { encoded: "//////////8=", byteOffset: 0, length: 8 },
  view: { index: 0, byteOffset: 0, length: 4 },
  self: "#",
  "$types": {
    big: "bigint",
    set: "set",
    exp: "regexp",
    map: "map",
    "map.0": "arrayNonindexKeys",
    data: "uint8array",
    view: "uint16array",
    self: "#"
  }
}

The footprint is much heavier than the ungap version, since it depends on a whole separate library. I’ve had issues importing it in various scenarios, which is why I’ve created this wrapper. That being said, it also has a few advantages:

• The format is human readable
• It can be consumed by endpoints that aren’t aware of the encoding:
  • It makes reasonable choices for representing non-JSON types, e.g. arrays for sets and arrays of tuples for maps
  • The $types property can be ignored entirely. If you were to put this in a JSONB column in Postgres or SQLite, it would not be out of place, work with builtin functions for the most part, and could even be updated without corrupting the format.
• It has support for some builtin types like Blob, File and FileList, but the use of -Async variants of each library function is required.

In addition to the issues mentioned above, it also has more fundamental problems:

• When encoding with MessagePack or similar, you pay a 33% Base64 tax for nested binary data
• It preserves the object identity of array buffers in typed array, meaning extra caution is necessary when dealing with array buffer views.
• The $types object duplicates the keys and can add significant overhead for deeply nested structures.
• No support for Error, BigUint64Array and BigInt64Array
• Defining custom types is only possible by opening up the underlying Typeson library
• Generally more quirky and less battle-tested

MessagePack + Extensions

Message Pack is a popular and simple binary format that is in many cases a drop-in replacement for JSON. While it inherits the semantics of JSON, it comes with support for extensions that some libraries use to implement the SCA out of the box. For example, msgpackr has a structuredClone option that matches the semantics of the SCA pretty accurately.

• It supports basic types and preserves (circular) references
• Typed arrays are treated as slices (the array buffer reference is not preserved)
• Unlike the options so far, it is a binary format that will efficiently inline binary data without overhead
• The extension system is open, i.e. developers can define additional types
• It has no support for platform objects like Blob

import { Packr, Unpackr } from 'npm:msgpackr';
const packr = new Packr({ structuredClone: true });
const unpackr = new Unpackr({ structuredClone: true });
unpackr.unpack(packr.pack(obj))

The obj above produces the following buffer:

d6 69 00 00 00 01 d4 72 40 97 a3 62 69 67 a3 73 65 74 a3 65 78 70 a3 6d 61 70 a4 64 61 74 61 a4
76 69 65 77 a4 73 65 6c 66 cf ff ff ff ff ff ff ff ff d4 73 00 93 01 02 03 d4 78 00 92 a8 5c 2f
5e 64 2b 24 5c 2f a1 67 81 a1 61 03 c7 09 74 01 ff ff ff ff ff ff ff ff c7 09 74 04 ff ff ff ff
ff ff ff ff d6 70 00 00 00 01

Note the 3 distinct blocks of ff that are the Uint64Max bigint, Uint8Array and Uint16Array respectively.

Problems arise when interacting with other MessagePack implementations. The extensions used to make the SCA work are not standardized and different implementations make different choices. For example, msgpackr uses extension type 0x78 (lowercase x) to denote a RegExp object, while msgpack-lite, another popular library, uses 0x0a for much the same purpose.

Besides mismatching extension codes, the concept of a “reference” isn’t standardized either and msgpackr’s SCA preset uses additional optimizations that will be difficult to replicate in other libraries.

For these reasons, MessagePack in general and msgpackr/msgpack-lite in particular are only good choices when controlling both ends of the serialization and both ends are written in JavaScript and have access to these libraries. For other languages, rather than porting these libraries, there’s probably more luck to be had with the following approach.

CBOR + Standard Extensions

While I can’t say I’m a big fan of CBOR, I have to admit that its centralized registry for extension types makes it a better exchange format for JS objects when interoperability is desired. Since CBOR is an integral part of Passkeys, the world isn’t entirely devoid of CBOR parsers either, raising this above the level of a theoretical benefit.

Specifically, the extensions relevant to the SCA are:

Late additions like 21066 for RegExp and the odd naming here and there make it rather obvious that this format wasn’t designed with the SCA in mind either, but the current sent of extensions are enough to mimic the behavior of the SCA reasonably well.

Thanks to its centralized registry and unlike your typical MessagePack library, there’s at least a chance that something reasonable will pop out the other end when two different CBOR implementations are used, making this the preferred choice when a binary format with interoperability and efficient inlining of nested buffers is desired.

For completeness sake, here is our trusty obj encoded as CBOR with the extensions above:

d8 1c b9 00 07 63 62 69 67 1b ff ff ff ff ff ff ff ff 63 73 65 74 d9 01 02 83 01 02 03 63 65 78
70 d9 52 4a 82 68 5c 2f 5e 64 2b 24 5c 2f 61 67 63 6d 61 70 d9 01 03 a1 61 61 03 64 64 61 74 61
d8 40 48 ff ff ff ff ff ff ff ff 64 76 69 65 77 d8 45 48 ff ff ff ff ff ff ff ff 64 73 65 6c 66
d8 1d 19 00 00

The cbor-x library was used to generate the above, with the following settings and additions:

import * as CBOR from 'npm:cbor-x'
const encoder = new CBOR.Encoder({ structuredClone: true, useRecords: false, pack: false, tagUint8Array: true, structures: null })
CBOR.addExtension({
  tag: 21066,
  Class: RegExp,
  encode(regexp, encode) { return encode([regexp.source, regexp.flags]) },
  decode(data) { return new RegExp(data[0], data[1]) },
});
encoder.encode(obj);

There are no registered types for platform objects like File or Blob. As with Errors, the generic “Serialised language-independent object with type name and constructor arguments” (tag 27) will have to do in these cases.

Conclusion

When needing a serialization format that lives within Node Deno or Bun (or possibly as an exchange format between them)³, the node:v8 module is a good and performant choice. However, it lacks documentation besides its source code in the V8 repository and I’m not aware of any alternative implementations⁴.

When a more open format with more interoperability is needed, a world of possibilities opens up. @ungap/structured-clone and @worker-tools/structured-json offer backwards compatibility with JSON. The latter is human-readable, while the former is mostly not. Both can be binary encoded with a variety of libraries like MsgPack, but suffer from overhead associated with nested buffers, ranging form 33% to ~50%. Structured JSON suffers form overhead with (deeply) nested objects in general.

Formats with binary serialization in mind from the beginning are MessagePack and CBOR, each requiring a set of extensions to achieve SCA compatibility. CBOR is the better choice when documentation and standardization are desired. It has registered extension codes for the core ECMAScript types. If it’s more important to have any library at all, MessagePack + non-standard extensions might be a better choice, but one library’s SCA implementation will not match another’s, most likely requiring additional user code. With only 127 extension codes to pick from, conflict freeness is not guaranteed.

Web scraping is getting increasingly difficult, ironically not least due to Cloudflare’s own Scrape Shield, which deploys various techniques such as TLS fingerprinting to determine who is accessing a site. CF Workers doesn’t hide the fact that it is not a User Agent (i.e. browser). It is only suitable for light scraping uses. Of course the same applies to any HTMLRewriter use case.

In this post we’ll be implementing a custom Hacker News API by scraping its HTML frontend, the same approach used by the unofficial HN API for Node. The examples are taken from Worker News.

Introduction

At first glace, HTMLRewriter is a poor fit for HTML scraping. It’s API is geared towards rewriting a HTML response, not extracting data from it. To familiarize ourselves with the API, here is a slightly modified example from Cloudflare’s tutorial:

addEventListener('fetch', ev => ev.respondWith(handleEvent(ev)))

async function handleEvent(ev) {
  const response = await getAssetFromKV(ev)
  return new HTMLRewriter()
    .on("[data-i18n-key]", {
      element(el) { // <-- Everything callback-based
        const i18nKey = el.getAttribute("data-i18n-key");
        const str = strings[i18nKey]
        if (str) el.setInnerContent(str)
      },
    })
    .transform(response) // <-- Returns a `Response`
}

First, we note that everything in HTMLRewriter is callback-based. Second, we note its transform API: It expects to turn one Response into another. When web scraping, we just want to consume a response but not process it in any further or send it to the client.

Besides these ergonomic inconveniences, its biggest drawback is its lack of “inner HTML” API. HTML Rewriter can notify us of element tags or text chunks, but it can’t give us the contents of an entire subtree in the DOM. There is hope that this will be implemented in the future, but for now we’ve have to work around this. The recently added (still undocumented) onEndTag feature finally gives us the tool to make this possible.

• Introduction
• Consuming a Response
  • Streaming Consume
• Extracting Data
  • Extracting Data Streams
• Extracting HTML Subtrees
• Appendix
  • Custom Event Polyfill

Consuming a Response

We first work around the transform issue. What makes the example above work is the fact that the Response provided by transform is passed to respondWith in the fetch event. This causes data to be pulled from the stream as it makes its way towards the (browser) client, which sets the whole streaming pipeline in motion.

Since we won’t be sending the craping response to the client, we need a different way to pull data from the stream. A quick and dirty solution is to just await .text() on the transformed response. But this causes the entire response body to be loaded into a string:

const response = await fetch('https://news.ycombinator.com')
if (!response.ok) throw Error('Scrape shield encountered!');

const rewriter = new HTMLRewriter()
  .on(".athing[id]", {
    element(el) { /* TODO */ }
  });

const _text = await rewriter.transform(response).text();

While this works, it’s still not ideal since it will force the entire document into memory at one point, only to be discarded right afterwards.

Streaming Consume

A better solution is to consume the response stream chunk by chunk:

async function consume(stream: ReadableStream) {
  const reader = stream.getReader();
  while (!(await reader.read()).done) { /* NOOP */ }
}

await consume(rewriter.transform(response).body!);

The consume helper function, as the name suggests, pulls every chunk from the stream and discards it. By accepting a readable stream we keep it generic enough to accept other types of readable streams as well. In the case of a Fetch API Response, we access its stream via the .body property.

Extracting Data

With the transform pipeline set in motion, we can focus turn our attention to the callbacks. Once again, we start with a quick and dirty solution (that may very well be good enough for your use case) and then improve it later.

In the code below we extract the Hacker News item id from every post on the landing page:

const ids = []
const rewriter = new HTMLRewriter()
  .on(".athing[id]", {
    element(el) {
      ids.push(el.getAttribute('id')!)
    }
  })

await consume(rewriter.transform(response).body!)

// `ids` is now populated:
console.log(ids)

We use good old imperative programming and async/await to populate our ids array. As I said earlier, this may very well be good enough for you, but it does have the drawback of consuming the entire response before we continue processing the ids. In other words, we lose the streaming aspect of HTMLRewriter.

Extracting Data Streams

A more fancy approach is to turn the callbacks into an async iterable that we process as data arrives in a for await loop. For a refresher on asynchronous data processing in JavaScript, see my own Async Generators in the Wild.

Turning a (multi-)callback API into an async iterable is not trivial. It involves two steps:

1. First we turn callback invocations into events,
2. then we use a utility function to turn the event stream into an async iterable.

The utility function is provided by yours truly as event-target-to-async-iter, but the code is simply an adaptation of Node’s on utility function with the Node-specific parts removed.

const target = new EventTarget(); // 1

const rewriter = new HTMLRewriter()
  .on(".athing[id]", {
    element(el) {
      target.dispatchEvent(new CustomEvent('data', {  // 2
        detail: el.getAttribute('id') 
      }));
    }
  })

const data = evenTargetToAsyncIter(target, 'data'); // 3

consume(rewriter.transform(response).body!) // 4
  .catch(e => data.throw(e)) // 5
  .then(() => data.return()) // 6

for await (const id of data) { // 7
  console.log(id)
}

There is a lot to unpack here. First of all, any experienced developer will undoubtedly spot the many of ways of making this more ergonomic, but for our purposes here I left it as verbose as is.

The goal is to process scraped data in (7) via for await loop. This leaves us with many opportunities down the line, such as streaming JSON for APIs, streaming HTML, Server Sent Events, etc…

While it is possible to dispatch events on the global scope (which implements EventTarget), it is advisable to use a new EventTarget as in (1) instead. Recent compatibility dates of CF Workers support this out of the box.

Unfortunately the same can’t be said for CustomEvent (2), but a minimal polyfill is trivially implemented. Custom Events are good use here, as the provide a generic detail property that we can use to store data. We fire them under the generic data event name. You can pick anything here, it only needs to match the key used in (3).

In (3) we turn the event target into an async iterable. Note that this only sets up queues and event listeners, but does not do anything by itself.

The process only starts once we start pulling data in (4). What’s important is that we do not await here! Doing so would defeat the purpose of setting up the streaming pipeline, as we wait for the entire response to be consumed (filling up the internal queues of eventTargetToAsyncIter) before continuing the execution.

Not awaiting a promise opens us up to the possibility of an unhandled exception, so we need catch it in (5) and forward it to the async iterable via throw(). This will cause the error to show up in (7) during for await looping.

Finally, in (6) we prevent for-await from getting stuck in an endless loop. Event targets, unlike async iterables, do not have a concept of an end, so we manually call return() on the iterable when the response stream is fully consumed.

Extracting HTML Subtrees

We make up for HTMLRewriter’s lack of innerHTML by combining two selectors and use of the recently added onEndTag API:

const commText = '';
const rewriter = new HTMLRewriter()
  .on('.fatitem .commtext', { // 1
    text({ text }) { commText += text }
  })
  .on('.fatitem .commtext *', { // 2
    element(el) { 
      const maybeAttrs = [...el.attributes].map(([k, v]) => ` ${k}="${v}"`).join('');
      commText += `<${el.tagName}${maybeAttrs}>`;
      el.onEndTag(endTag => { 
        commText += `</${endTag.name}>`;
      });
    }
  })

If we only used the first selector and applied it to, e.g. this comment we would get the following:

There's lots of things going on this this space. It seems every other day I discover another 
Cloudflare Workers-like implementation (granted, most of them are for testing/development). 
I'm cataloging them here for anyone who's interested: https://workers.js.org

This seems correct at first, but it is missing the <a> tag on the link. This works because the text callback delivers every text chunk in the entire subtree. It does however ignore all the tags.

Once we add the second selector with the extra *, we are notified of all opening and closing tags in the entire subtree and can append them to the string. Because HTMLRewriter is a stream processor internally, we can expect these callbacks to be called in the correct order.

Appendix

Custom Event Polyfill

Note that this is by no means a spec-compliant implementation of CustomEvent, but it works for our purpose here.

if (!('CustomEvent' in self)) {
  class CustomEvent<T = any> extends Event {
    readonly detail: T; 
    constructor(event: string, { detail }: CustomEventInit<T>) { 
      super(event); 
      this.detail = detail as T;
    }
  }

  Object.defineProperty(self, 'CustomEvent', {
    configurable: false,
    enumerable: false,
    writable: false,
    value: CustomEvent
  });
}

font-family: system-ui, 
             -apple-system, BlinkMacSystemFont, 
             "Segoe UI", 
             "Roboto", 
             "Oxygen", 
             "Ubuntu", 
             "Cantarell", 
             "Fira Sans", 
             "Droid Sans", 
             "Helvetica Neue", 
             Arial, sans-serif;

Before we get to the monospaced part, it’s worth going through each font to see where its comes from and what the monospace equivalent should be:

The Monospaced System UI CSS Font Stack

For developers it would be nice to have a monospaced equivalent, but so far nobody seems to have assembled the necessary fonts. Given what we learned above, we can lookup the default monospace fonts for each platform, which gives us:

font-family: ui-monospace, 
             Menlo, Monaco, 
             "Cascadia Mono", "Segoe UI Mono", 
             "Roboto Mono", 
             "Oxygen Mono", 
             "Ubuntu Mono", 
             "Source Code Pro",
             "Fira Mono", 
             "Droid Sans Mono", 
             "Consolas", "Courier New", monospace;

While most of these are pretty self-explanatory, i.e. the “Mono” or “Monospace” version of the same font family, some are worth expanding on:

ui-monospace

Future system monospace font⁸. Already supported in Safari 13.1⁹. This is also the only way to access macOS’ new ‘SF Mono’ typeface.

Menlo, Monaco

Default in older versions of macOS / OSX, but also used in browsers that don’t support ui-monospace on macOS.

Cascadia Code, Segoe UI Mono

These fonts are intended for Windows, but neither is included by default¹⁰. Windows does not have a default monospace font (unless you count ‘Courier New’). However, ‘Segue UI Mono’ has been shipped with some Microsoft software, so it’s worth including here.
‘Cascadia Mono’ is a new monospace font Microsoft is working on that ships with Windows Terminal¹¹, and is therefore likely to be installed by a more technical audience.

Source Code Pro

Default in GNOME¹², but from a different family as the default variable-width typeface ‘Cantarell’

This is known as “Web 2.0”. It is a shockingly simple model that can still be used today, even without a framework. Really, all you need is to send some text to the browser.

While such apps might not quite rise the expectations of modern, mobile app-aware audiences, they trivially beat SPWA⁠s in terms of memory consumption, energy efficiency, and reliability. The path to superior time to first byte, time to first paint, and time to interactive is also much clearer.

I’ve recently built just such an app, and I can say that I’m quite pleased with the result. Interestingly, I didn’t set out to write a Web 2.0 app, I’ve just wanted to quickly throw together a dashboard and things went from there. I expected this to be temporary; to be replaced by a full React app later on. Now I think I’m just going to keep the HTML version.

The Complexity of Single Page Web Apps

When combined with data fetching, modern SPWA⁠s easily devolve into an exercise in ad-hoc distributed systems engineering. The need for ever more sophisticated “easy-to-use” libraries arises to help manage all this extra state and asynchronicity.

Writing Plain Old Web Sites can be an eye opener. You thought you needed all this complexity, but actually what you’re giving up is mostly cosmetic.

I’ve stated some benefits above, but there are more:

• Simplified data fetching — Pulling all the strings together in a central location before generating a single document is a simpler model than fetching from many sources at different times and updating the UI in patches
• No need for crazy build tool chains — The size of the Create React App tool chain is well known. Other frameworks take it a step further and introduce their own compiler, etc.

What You Lose

Obviously, most web apps cannot be built in this way. Anything with a high degree of interactivity, real-time, etc. is a very bad fit. As a rule of thumb, the closer an app resembles a series of documents, the better this model fits. This should not be surprise, given the origins of the web.

If there was a vibrant ecosystem of Web Components, some of the interactive capabilities could be added back, but this is purely hypothetical at this point. The available stock of Web Components is underwhelming, while all the best web components (lower case) are written in React.

Could There Be a Comeback of POWAs?

Probably not, but there are some trends that are helping:

• Edge computing moves the server closer to the visitor, which reduces latency. I’ve used Cloudflare Workers for my dashboard, and response times can be crazy fast. More on that in an upcoming post.

• “JavaScript fatigue” is wearing out developers and they might be inclined to look back to a simpler time. They will find it in POWA⁠s, but not without losing many capabilities (see above)

• The ad industry. The use of ad blockers is already prevalent. What if web users took it a step further and started disabling JavaScript altogether? Many websites no longer work without JavaScript, but those that do usually have their usability improved. For some publications, disabling JavaScript effectively circumvents the pay wall.

• Apple. It would be a very Apple thing to do to give user an easy option to disable JavaScript. It would be called “Enhanced Privacy Mode” or “Request permission to execute untrusted code”. If you think that would never happen, remember the history of Adobe Flash.

There’s obviously precedent for “Everything Old is New Again” in user interface development, and React itself is an example of that. While a comeback for plain old web apps seems unlikely, crazier things have happened.

I’ve enjoyed writing a web app using pretend-1998 technology, and I’m looking forward to writing another one should a situation come up. At the same time, there’s still an obvious, battle-tested, and straight-forward choice for most web apps today, which is React.

• Overview
• Async Method Design
• Lazy-Initialization
• Conclusion
• Appendix
  • Why use classes?
  • Master Design
  • Typestate Pattern
  • Factory Functions

Overview

A pattern that I’ve seen is to have an asynchronous init function that does most of the initialization, but I think that’s a bad idea:

• Am I going to call init() more than once? In fact, what if I do?
• Didn’t I already express my intent of initializing by saying new?
• Bonus: Can I write it as a one-liner?

For these reasons, what I prefer is the following:

const b64 = await new Base64Encoder().initialized;

It might not be obvious, but this is valid JavaScript — no extra parenthesis required. But how does this work? Let’s look at the class definition.

I’m using (not yet finalized) private fields syntax for brevity, for production code I recommend using the WeakMap pattern instead.

class Base64Encoder {
  #instancePromise;
  #instance = null;

  // No async constructors possible, but we can use an IIAFE for async code.
  // Note that the return type of any async function is a promise, which we can store.
  constructor() {
    this.#instancePromise = (async () => {
      const response = await fetch('./base64.wasm');
      if (!response.ok) throw Error('...');
      const arrayBuffer = await response.arrayBuffer();
      return WebAssembly.instantiate(arrayBuffer); // no await
    })();
  }

  // Again, no async getters are possible, but we can return a promise.
  // Using a getter to ensure `readonly`-ness
  get initialized() {
    return this.#instancePromise.then(({ instance }) => {
      this.#instance = instance; // store the result
      return this; // this is what makes the one-liner possible!
    });
  }

  encode(data) {
    if (!this.#instance) throw Error("Didn't you forget something?");
    // Do something with `#instance`...
  }
}

This achieves our design objectives:

• The initialization starts right away
• We can’t call the initialize code more than once
• We can create a new object in one line with a de-facto async constructor.

This is made possible by two key components:

• A constructor that assigns a promise to a local variable (via an IIAFE) and
• a getter that returns a promise that resolves to this. The getter ensures that the property is readonly.

A lot is going on during initialization. Looking at it again with added parenthesis, the order should be clearer:

const b64 = await ((new Base64Encoder()).initialized);

1. A new object instance is created that starts the (async) initialization immediately
2. The initialized getter returns a promise that eventually resolves to the newly created instance.
3. The event loop takes over
4. ???
5. The initialization code finishes, “unblocking” the user code

While I wouldn’t recommend it due to error-proneness, initialization can also be split in two:

const b64 = new Base64Encoder();
// ...
await b64.initialized;
// ...
const str = b64.encode(/*...*/)

Async Method Design

The design above has one major problem: If the caller forgets to await the initialized promise, calls to the API methods will fail. Worse: It might work depending on the timing, effectively introducing a race condition EDIT: Nope, this is taken care of by the assignment to the #instance field.

There is a another solution that does not require any kind of initialization — at least none that the caller is aware of — but it requires that the methods themselves are async, even though they wouldn’t have to be otherwise:

class Base64Encoder {
  #instancePromise;

  constructor() {
    this.#instancePromise = (async () => {
      const response = await fetch('./base64.wasm');
      if (!response.ok) throw Error('...');
      const arrayBuffer = await response.arrayBuffer();
      return WebAssembly.instantiate(arrayBuffer);
    })();
  }

  async encode() { //!!4
    const { instance } = await this.#instancePromise;
    // Do something with `instance`...
  }
}

The main idea here is that we await the initialization promise at the beginning of every method call. It’s important to understand that the initialization code only runs once. Once a promise is resolved, we can await it as many times as we want, and it will resolve instantly (but there might be some minor overhead associated with the creation of a Promise behind the scenes).

Now the API pattern as changed to

const b64 = new Base64Encoder()
const encoded = await b64.encode(/*...*/)

Lazy-Initialization

Even though I stated in the beginning that immediate initialization is a design goal, one could reasonably prefer to delay initialization to a later point. The aforementioned async init() method will work in that case, but we could also delay it to the last possible moment: The first method call. Expanding on the async method design from before, we only have to make a minimal change:

class Base64Encoder {
  #instancePromise = null;

  async encode() { //!!11
    this.#instancePromise = this.#instancePromise || (async () => {
      const response = await fetch('./base64.wasm');
      if (!response.ok) throw Error('...');
      const arrayBuffer = await response.arrayBuffer();
      return WebAssembly.instantiate(arrayBuffer);
    })();

    const { instance } = await this.#instancePromise;
    // Do something with `instance`...
  }
}

Ruby developers have seen this pattern before in the form of the ||= operator. The result is the same: We assign to #instancePromise only once, and repeated invocations of encode are waiting for the same promise to resolve.

Conclusion

In this post I’ve showed three design patterns that deal with async initialization and demonstrated them on a real-world example. I’ve shown which pattern I prefer. While writing this post, I’ve stumbled on several more ideas that you can find below. There, you’ll also find your favorite pattern from that book I didn’t read.

Appendix

Why use classes?

I’m not a fan of classes but I will use them where they make sense, which is the case here because:

• Node docs recommend isolating state for its upcoming ES modules support, for reasons explained in the linked article.

• For the WASM Base64 example above, it’s a good idea to give the user a hand in managing memory consumption, if only indirectly.

  Specifically, the implementation might need to grow the WebAssembly memory to fit the data. Without attaching the WebAssembly instance to an object, the memory could never be garbage collected. We could discard the instance ourselves after each call, but that would prevent legitimate cases where the caller might want to use the same instance multiple times. I suspect this is also the reason why TextEncoder and TextDecoder are designed as classes and not functions.

Master Design

The following code combines all 3 approaches outline above + the manual call to init() that I’ve mentioned in passing. While I said earlier that I don’t like the init pattern, note that the implementation below is protected against multiple invocations.

class Base64Encoder {
  #instancePromise = null;
  #instance = null;

  async init() {
    this.#instancePromise = this.#instancePromise || (async () => {
      const response = await fetch('./base64.wasm');
      if (!response.ok) throw Error('...');
      const arrayBuffer = await response.arrayBuffer();
      return WebAssembly.instantiate(arrayBuffer);
    })();
    const { instance } = await this.#instancePromise;
    this.#instance = instance;
  }

  get initialized() {
    return this.init().then(() => this);
  }

  encode() {
    if (!this.#instance) throw Error("Didn't you forget something?");
    // ...
  }

  #promises = {
    async encode(data) {
      await this.init();
      const { instance } = await this.#instancePromise;
      // ...
    },
  };

  get promises() {
    return this.#promises;
  }
}

Typestate Pattern

When using TypeScript (but works with vanilla JS too), the initial design can be extended to include an additional class called Base64EncoderInitialized or similar that implements the encode method, while removing it from the original. Adapting the initialized promise so that it resolves to an instance of the new class gives us the static guarantee that encode can only be invoked after initialization is completed.

get initialized() {
  return this.#instancePromise.then(({ instance }) => new Base64EncoderInitialized(instance))
}

Factory Functions

Instead of working around constructor and getter limitations, we can just define a static async function that does the initialization work and returns the new instance. We can spice it up by using a private symbol to prevent callers from creating uninitialized instances via the constructor (JS’ version of private constructors):

const CREATE = Symbol('create');

class Base64Encoder {
  static async create() {
    const obj = new Base64Encoder(CREATE);
    // Do async initialization here
    return obj;
  }

  constructor(token) {
    if (token !== CREATE) {
      throw Error("Base64Encoder can't be created via constructor, use Base64Encoder.create instead");
    }
  }
}

// Usage
const base64 = await Base64Encoder.create();

• Basic Due Diligence
• Replacing CryptoJS
  • Overview
  • The CryptoJS Cipher Prefix
  • Dangerously Deriving Parameters
  • The EVP Key Derivation Function
• Usage

Basic Due Diligence

CryptoJS is a popular library for doing cryptography in the browser. After doing some basic research, I’m convinced that it is not a production-quality cryptography library. Not by a long shot.

What I found is a library of quasi-unknown origin, maintained by an amateur, unsafe default parameters, no bug bounty program, no published audit of any kind, and no discernable source of income, otherwise known as The Greatest Hits of Soon to be Broken Cryptography Libraries¹.

Replacing CryptoJS

In this post we’ll be replacing CryptoJS’ AES module with the Web Cryptography API. The biggest challenge when phasing out CryptoJS is dealing with data previously encrypted by it. We can keep the dependency around for that purpose, but personally I’d rather delete it immediately and use standard APIs for decrypting old data instead.

CryptoJS uses the standard AES-CBC algorithm which also ships as part of the Web Cryptography API. Web Crypto only includes a single padding scheme for non-block-sized payloads, but it’s the same one used by CryptoJS by default.

However it gets more complicated with respect to key derivation. A peak under the hood reveals that the algorithm to derive a key from the passphrase is not standardized and therefore not part of Web Crypto, and it uses the broken MD5 hash that is also not part of Web Crypto. Unfortunately, we’ll have to support this in order to decrypt old data, but we can take some measures to prevent make it harder to abuse.

Overview

Our high level function is going to look like this:

async function decryptCryptoJSCipherBase64(cryptoJSCipherBase64, password, { 
  keySizeInt32 = 256 / 32,
  ivSizeInt32  = 128 / 32,
  iterations   = 1,
} = {}) {
  const { salt, ciphertext } = parseCryptoJSCipherBase64(cryptoJSCipherBase64);
  const { key, iv } = await dangerouslyDeriveParameters(password, salt, keySizeInt32, ivSizeInt32, iterations);
   
  const plaintextArrayBuffer = await crypto.subtle.decrypt({ name: "AES-CBC", iv }, key, ciphertext);
  return new TextDecoder().decode(plaintextArrayBuffer);
}

A couple of notes:

• The default parameters are taken from CryptoJS.
• I assume the cleartext is a string. If not you’ll have to remove the text decoding part.
• We accept the cipher in base64 because that’s the default returned by CryptoJS’ toString function. You’ll have to adjust the code slightly to accept other formats.
• I’m using verbose variable names because it’s really easy to mess up types and encodings otherwise.
• All sizes in this article are given in multiples of 32-bit. I chose this unit because it is also used by CryptoJS, making it easier for the fast-moving npm-hacker to copy-paste the code and get it to work. To communicate this admittedly odd choice to everyone else, I borrow the DWORD designation from Microsoft where it is commonly known to be 32 bits I’ve updated the code to use the Int32 designation instead of DWORD.
• I follow the React-tried-and-true practice of giving dangerous functions a verbose and uncomfortable name.

The CryptoJS Cipher Prefix

Next we look at parsing the input ciphertext. CryptoJS prefixes the ciphertext with Salted__ (exactly 64 bits), followed by a 64-bit salt.

+----------+----------+------------------------
| Salted__ |  <salt>  | <ciphertext...
+----------+----------+------------------------
|  64 bit  |  64 bit  | variable length

To retrieve the salt and ciphertext as Uint8Arrays we use the following. I should note that the following code is mostly extracted from CryptoJS and rewritten using modern JS idioms and APIs, specifically typed arrays.

const HEAD_SIZE_INT32 = 2;
const SALT_SIZE_INT32 = 2;

function parseCryptoJSCipherBase64(cryptoJSCipherBase64) {
  let salt;
  let ciphertext = base64ToUint8Array(cryptoJSCipherBase64);

  const [head, body] = splitUint8Array(ciphertext, HEAD_SIZE_INT32 * 4);

  // This effectively checks if the ciphertext starts with 'Salted__'.
  // Alternatively we could do `atob(cryptoJSCipherBase64.substr(0, 11)) === "Salted__"`.
  const headDataView = new DataView(head.buffer);
  if (headDataView.getInt32(0) === 0x53616c74 && headDataView.getInt32(4) === 0x65645f5f) {
    [salt, ciphertext] = splitUint8Array(body, SALT_SIZE_INT32 * 4);
  }

  return { ciphertext, salt };
}

I’ll provide the helper functions at the end, but given how verbose the names are, it’s suffice to say that they do exactly what they say they do.

Notice the multiplications by 4 to go from Int32s to bytes, and the offset of 4 on getting the second Int32 value for the same reason.

Dangerously Deriving Parameters

Next we shift our attention towards the enigmatic dangerouslyDeriveParameters function. This is where we take a cryptographically weak passphrase and turn it into a supposedly strong cryptographic key. Given the default parameters this is not actually the case.

async function dangerouslyDeriveParameters(password, salt, keySizeInt32, ivSizeInt32, iterations) {
  const passwordUint8Array = new TextEncoder().encode(password);

  const keyPlusIV = dangerousEVPKDF(passwordUint8Array, salt, keySizeInt32 + ivSizeInt32, iterations);
  const [rawKey, iv] = splitUint8Array(keyPlusIV, keySizeInt32 * 4);

  const key = await crypto.subtle.importKey("raw", rawKey, "AES-CBC", false, ["decrypt"]);

  return { key, iv };
}

• Note how both the key and IV are derived from the password. I think this would be okay if the KDF wasn’t so weak and the salt is random and unique, but this is not something the Web Cryptography API’s deriveKey function would support if it implemented the EVPKDF.
• We only allow the key to be used for decryption. This is to prevent accidental or intentional use for encryption, for which it is too weak.

The EVP Key Derivation Function

Next we turn to the dangerousEVPKDF function. It is the same key derivation function used by OpenSSL (google EVP_BytesToKey for details) and not dangerous by itself, but in the case of hard-coding MD5 as its hash function it certainly is.

This is the only part where we rely on an external dependency, because MD5 is not provided by the Web Cryptography API. I’m using js-md5 here because it supports array buffers, but did not do any research on it otherwise. I figured there’s no such thing as a safe MD5 implementation anyway. You have been warned.

import * as md5 from 'js-md5'

function dangerousEVPKDF(passwordUint8Array, saltUint8Array, keySizeInt32, iterations) {
  let derivedKey = new Uint8Array();
  let block = new Uint8Array();

  while (derivedKey.byteLength < keySizeInt32 * 4) {
    block = md5.arrayBuffer(concatUint8Arrays(block, passwordUint8Array, saltUint8Array));

    for (let i = 1; i < iterations; i++) {
      block = md5.arrayBuffer(block);
    }

    derivedKey = concatUint8Arrays(derivedKey, new Uint8Array(block));
  }

  return derivedKey;
}

Note that this function could be made much more time and space efficient by pre-allocating and reusing typed arrays. However, it was easier to get it right this way, and it should also be easier to follow along.

Usage

That’s it. We can now decrypt data perviously encrypted with CryptoJS at a fraction of the code size, mostly asynchronously, and mostly using fast native code.

import AES from 'crypto-js/aes';

const cleartext = "This is a message to be encrypted and decrypted";
const password = "passw0rd!";
const cryptoJSCipherBase64 = AES.encrypt(cleartext, password).toString();

decryptCryptoJSCipherBase64(cryptoJSCipherBase64, password).then(x => {
  console.log(x);
  console.log(x === cleartext);
});

As promised, here is the link to the full example including utility functions. Keep in mind that these were not written with performance in mind and could all be improved.

In Part II we will deal with deriving a stronger key from the same passphrase and encrypting the data again. This is mostly run-of-the mill Web Crypto code and examples are readily available on the web. However, there’s some specialty involved with respect to dealing with the Salted__ prefix and integrating this nicely with the framework we’ve established in this post.

However, sometimes the tool you know best is best for the task at hand. Since I’ve been mostly using JS recently, I’ve felt more productive using it than any other language, even though I had to write my own functional programming library.

• Iterator Operators
• Infinite Sequences
• A Little Iterator Library
• Further Reading

Iterator Operators

Thanks to generator functions, operators can be implemented quickly and concisely. Not only is it a good learning experience, but it also gives you full knowledge of what’s happening in the background. For example, the ubiquitous map functions is easily implementas as:

function map(f) {
    return function* (xs) {
        for (const x of xs) yield f(x);
    }
}

Note that the inner function is a generator function (indicated by the *) that takes an iterable and returns an iterable. The reason why currying is applied here is to make it usable with a pipe function, so that we can write pipe(x, f, g, h) instead of f(g(h(x))):

const checksum = pipe(
    ids,
    map(id => frequencies(id)),
    map(fqs => [
        pipe(fqs.values(), some(x => x === 2)),
        pipe(fqs.values(), some(x => x === 3)),
    ]),
    unzip2(),
    map(xs => pipe(xs, map(x => x ? 1 : 0), sum())),
    reduce((a, b) => a * b, 1),
);

There is a lot going on here — and yes, this is a working solution for Day 2 — but the point is to show that there is a functional programming style that’s made possible by iterators and generators. Each of these operators has an implementation that’s similar to map in both length and complexity.

What’s nice about these iterator-based operators is that intermediate results are never fully realized in memory, e.g. in the example above, at no point is there an array of all frequencies — just a generator with some local state that knows how to produce the next value.

By comparision, an implementation based on Array-methods would look something like this and produce an array of Maps and then another array of tuples, each of which is fully realized in memory, before proceeding to the next step.

ids
    .map(id => frequencies(id))
    .map(fqs => [
        [...fqs.values()].some(x => x === 2),
        [...fqs.values()].some(x => x === 3),
    ]);

Infinite Sequences

Iterator-based processing is especially handy when dealing with infinite sequences. For example, cycle repeats the provided sequence indefinitly:

const res = pipe(
    cycle(input),
    scan((a, b) => a + b, 0),
    scan((seen, freq) =>
        seen.has(freq)
            ? { reduced: freq }
            : seen.add(freq),
        new Set(),
    ),
    find(({ reduced }) => reduced)
);

What causes the above sequence to come to a stop is find. Once an element is found that meets the condition,the underlying implementation is no longer calling next on the generator up the chain, thereby ending the sequence. Obviously, at no point is an infinite sequence realized in memory.

Just as with map, the implementation of find is rather compact:

function find(p) {
    return function (xs) {
        for (const x of xs) {
            if (p(x)) return x;
        }
        return null;
    }
}

Note that next is implicitly being called as part of the for-of loop.

To show the full awesome might of these oparators, here is an almost complte solution for Day 10:

const RE = /position=<(.*), (.*)> velocity=<(.*), (.*)>/;

const [positions, velocities] = pipe(
    lines,
    map(line => RE.exec(line).slice(1).map(Number)),
    map(([a, b, c, d]) => [[a, b], [c, d]]),
    unzip2(),
    map(it => [...it]),
);

const [[finalPositions], sec] = pipe(
    constantly(velocities),
    scan((positions, velocities) => [...pipe(
            zip(positions, velocities)
            map(([[x, y], [dx, dy]]) => [x + dx, y + dy]),
        )],
        positions,
    ),
    pairwise(),
    zipWith(range(1)),
    find(([[prev, curr]]) => {
        const prevSize = calcSize(getBounds(prev));
        const currSize = calcSize(getBounds(curr));
        return currSize > prevSize;
    }),
);

Note that sometimes we want to realize a sequence in memory on purpose (achieved by [...x]). Unlike arrays, iterators can only be traversed once, which causes problems when trying to use a result in multiple places. This is one of the major differences between array- and iterator-based processing: With the later, one has to pay attention to the difference between iterators and iterables.

Maybe this is the reason why iterator-based functional programming reamains unpopular in JavaScript. The realiance on a pipe function to make complex pipelines possible is another. Hopefully this will become more practical with the introduction of a pipe operator.

In the meantime, take comfort in the fact that the implementation of pipe is really short as well:

function pipe(x, ...fs) {
    let res = x;
    for (const f of fs) {
        res = f(res);
    }
    return res;
}

A Little Iterator Library

I’ve since turned the patterns here into a separate project called lilit (Little Iterator Library), which comes in both an synchronous and asynchronous version. It’s still early but you can try it out like this:

Further Reading

• lilit — My little iterator library.
• IxJS — A functionl library similar to RxJS, using iterators (archieved)
• Ramda — A functional programming library for JS, using arrays
• What is the difference between iterables and iterators
• Advent of ES — My AoC 2018 solutions using iterators

• What are Async Iterators?
• Recursively Reading All Files From a Directory
• Consuming Async Iterators
• Stream-Like Processing

What are Async Iterators?

Async iterators are an upcoming feature of JavaScript that fills an important language gap: What happens when you combine an async function with a generator function?

Async functions return promises, generator functions return iterators, so what does an async generator function return? An iterator of promises!

An async generator function is declared with both the async and function* keywords:

const timeout = t => new Promise(r => setTimeout(r, t));

async function* foo() {
  yield 1
  await timeout(1000);
  yield 2;
  await timeout(1000);
  yield 3;
}

Sweet! So how can we use for something… useful?

Recursively Reading All Files From a Directory

Conceptually, all the files in a directory are iterable. However, in order to find them we’ll have to perform asynchronous operations on the file system. Unless we want to do all the work upfront, we’re better off with an async iterator that returns a result whenever the necessary async operations complete.

The implementation below works in node 11+ without any additional flags:

const { resolve } = require('path');
const { readdir, stat } = require('fs').promises;

async function* getFiles(rootPath) {
  const fileNames = await readdir(rootPath);
  for (const fileName of fileNames) {
    const path = resolve(rootPath, fileName);
    if ((await stat(path)).isDirectory()) {
      yield* getFiles(path);
    } else {
      yield path;
    }
  }
}

Similar to normal generator functions, we can use the yield* keyword to defer to another (async) generator, enabling an elegant, recursive implementation of recursive directory search.

It’s efficient in the sense that it only looks as far into the directory as you ask it to. Unlike observables, async iterators are pull-based and only run when the next method on the underlying async iterator is called.

Consuming Async Iterators

A somewhat awkward way of using async iterators — but without special language features — is like this:

function forAwait(asyncIter, f) {
  asyncIter.next().then(({ done, value }) => {
    if (done) return;
    f(value);
    forAwait(asyncIter, f);
  });
}

forAwait(getFiles('.'), x => console.log(x));

As we can see, async iterators really are just iterators that return promises.

While researching this post, I was surprised to find out that the above code will not exceed the call stack limit, suggesting that modern JavaScript engines support tail recursions!?

However, the idiomatic way of consuming async iterators is via the new for-await-of loop. The following code produces the same result as the code above:

for await (const x of getFiles('.')) console.log(x);

In order for for await to work, it has to be wrapped inside an async function, i.e. (async () => { /* ... */ })(). When trying this out in the node REPL, note that for await is not covered by the --experimental-repl-await flag, so the IIAFE is still necessary.

We can upgrade our forAwait function to become sort of an asynchronous reduce:

async function reduce(asyncIter, f, init) {
  let res = init;
  for await (const x of asyncIter) {
    res = f(res, x);
  }
  return res;
}

With our new reduce function, we can “consume” an entire async iterator and push the results into an array:

const toArray = iter => reduce(iter, (a, x) => (a.push(x), a), []);

const files = await toArray(getFiles('.')); 

This is similar to a traditional getFiles function that crawls an entire directory and returns all the files in a promise (see [my original answer on StackOverflow).

Stream-Like Processing

If we’re about to process the files in some way, why wait until we’ve crawled the entire directory before we continue? With async iterables we can run code as soon as the results come in:

// Turn each file name into an object with a `name` property
async function* toObj(filesIter) {
  for await (const name of filesIter) yield { name };
}

// Add the file size (which is another async operation)
async function* addFileSize(objIter) {
  for await (const obj of objIter) {
    const size = (await stat(obj.name)).size;
    yield { ...obj, size };
  }
}

// Compose functions
const processed = addFileSize(toObj(getFiles('.')))

// Pull values from the pipeline
for await (const { name, size } of processed) {
  console.log(`${name} (${size} bytes)`);
}

To be fair, the code above is not very real-worldy. Also, the a(b(c()))-style function wrapping is not very readable. Hopefully either a function bind operator or a pipe operator is going to make this more practial in the future.

For now, this is how async iterators can be processed and combined to form new async iterators. It is similar to how stream processing works. In fact, both node and browser streams (are about to) implement the async iterator interface so they can be consumed in this way.

For more on iterator processing in JavaScript, check out my new post Solving Advent of Code Puzzles with ES Generators.

└── user
    └── item
        ├── 1
        ├── 2
        ├── 3
        └── 4

If you don’t want to create extra index pages, use - instead. E.g. user-item-4. There is nothing inherently better about using / instead of -, but the later comes without the implicit suggestions that there is an underlying folder.

├── user-item-1
├── user-item-2
├── user-item-3
└── user-item-4

This may not matter for consumer-facing sites, but there’s plenty of developer sites that get this wrong, too:

• GitHub will give you URLs like /qwtel/hydejack/tree/v8, but /qwtel/hydejack/tree doesn’t exist.

  └── qwtel
      └── hydejack
          └── tree   # THIS FOLDER DOES NOT EXIST
              └── v8 # WE PUT THINGS IN IT ANYWAY
  

  

  Uhm, actually, this is the page I am looking for…

• npm will give you organization-namespaced package URLs like /package/@platformparity/streams, but the organization pages don’t exist.

  For some extra fun try /package, which — instead of returning a list of all packages — will redirect to /package/package 🙄

  └── package             # symbolic link
      ├── package         # with folders in it
      └── @platformparity # some of which don't exist
          └── streams     # but have stuff inside anyway 🤷‍♂️
  

  

  Now this is not the web page I am looking for.

• Jekyll’s default permalink structure looks like /2018/06/06/urls-are-directories, but no pages for /2018, 2018/06, and /2018/06/06 are created. The proper URL is /2018-06-06-urls-are-directories

To recap: / has a special meaning in path names, while - does not. When you need to visually separate parts of your URL, but have no plans of introducing extra index pages, use -.

However, we can hack our way around cluttering the history too much, using a few quick-and-dirty npm scripts.

For this post we assume that we want to check-in the dist folder only on tagged commits, which we create with the npm version command.

First, we add /dist to our .gitignore file, which tells git to ignore the top-level dist folder.

In package.json we add the following scripts:

{
  "scripts": {
    "preversion": "sed -i '' 's:/dist:#/dist:' .gitignore",
    "version": "npm run build && git add .",
    "postversion": "sed -i '' 's:#/dist:/dist:' .gitignore && git rm --cached -r dist && git add . && git commit -m 'restore pre-version .gitignore'"
  }
}

The sed command allows us to run a quick regex search-and-replace on a file.¹ I used : as a delimiter (instead of /) to make writing regexes that involve file paths easier.

The first regex looks for the dist entry and replaces it with something we can find later (we prepend a # so the file remains a valid .gitignore). The second regex does the reverse.

The execution order is as follows:

1. npm runs the preversion script. We remove the dist entry from .gitignore.²
2. npm bumps the version number and runs the version script. We build the project and add the files to git.
3. npm commits the files and tags the release, then runs the postversion script. We add the dist entry back to .gitignore and remove the build files from git in a follow-up commit.

The git history for every tagged release will look like this:

* ac58cf4 2018-04-06 | restore pre-version .gitignore (HEAD -> master)
* a8989f1 2018-04-06 | 1.2.3 (tag: v1.2.3)

This procedure still adds a lot of clutter to to the git history, but at least it is limited to tagged commits, and we aren’t at risk of checking in (development-) artifacts by accident.