express-asset-file-cache-middleware

A modest express.js middleware to locally cache assets (images, videos, audio, etc.) for faster access and proxying, for use in e.g. Electron apps.

For offline use of dynamic assets, e.g. in your Electron app or local express server.

const express = require("express");
const fileCacheMiddleware = require("express-asset-file-cache-middleware");

const app = express();

app.get(
  "/assets/:asset_id",
  async (req, res, next) => {    
    res.locals.fetchUrl = `https://cdn.example.org/path/to/actual/asset/${req.params.asset_id}`;

    res.locals.cacheKey = `${someExpirableUniqueKey}`;
    next();
  },
  fileCacheMiddleware({ cacheDir: "/tmp", maxSize: 10 * 1024 * 1024 * 1024 }),
  (req, res) => {
    res.set({
      "Content-Type": res.locals.contentType,
      "Content-Length": res.locals.contentLength
    });
    res.end(res.locals.buffer, "binary");
  }
);

app.listen(3000);

It works by fetching your asset in between two callbacks on e.g. a route, by attaching a fetchUrl onto res.locals. When the asset isn't cached on disk already, it will write it into a directory specified by the option cacheDir. If it finds a file that's alread there, it will use that.

On a cache miss the response body is streamed straight to disk (instead of being buffered entirely in memory), written to a temporary file and then atomically renamed into place, so an interrupted or partial download can never leave a corrupt cache entry. Non-2xx upstream responses (401/403/404/429/5xx, …) are never cached - the upstream status is proxied back to the client so the entry self-heals on the next request.

Since 1.4.0 the middleware supports HTTP Range requests, which is what browsers use for <video>/<audio> timeline seeking. Use the bundled sendBuffer helper as your final handler to get Range-aware responses for free:

app.get(
  "/assets/:asset_id",
  async (req, res, next) => {
    res.locals.fetchUrl = `https://cdn.example.org/path/to/actual/asset/${req.params.asset_id}`;
    next();
  },
  fileCacheMiddleware({ cacheDir: "/tmp" }),
  fileCacheMiddleware.sendBuffer
);

sendBuffer emits Accept-Ranges: bytes, returns 206 Partial Content with a Content-Range header when the client sends a valid Range header, and falls back to 200 OK with the full body otherwise. Unsatisfiable ranges return 416 Range Not Satisfiable.

The middleware also now sets Accept-Ranges: bytes before calling next(), so even consumers that keep their own res.end(res.locals.buffer) response handler will get browser seek UI enabled (they still won't serve 206s, though — you need sendBuffer for that).

The asset's contentType and contentLength are stored base64 encoded in the filename, thus no offline database is necessary

Note that setting cacheKey and cacheDir isn't strictly necessary, it will fall back to res.local.fetchUrl and path.join(process.cwd(), "/tmp"), respectively.

Since 1.6.0 the fetch+cache core is exported on its own, so callers that aren't serving an HTTP request (an Electron main process, a CLI precache, a queue worker) get the same streaming download, atomic rename, LRU accounting and progress reporting without standing up a local server:

const { cacheAsset } = require("express-asset-file-cache-middleware");

const result = await cacheAsset("https://cdn.example.org/big-video.mp4", {
  cacheDir: "/tmp",
  cacheKey: someExpirableUniqueKey, // optional, falls back to the URL
  onProgress: ({ received, total }) => {
    mainWindow.webContents.send("download:progress", { received, total });
  }
});

if (result.status === "cached") {
  console.log(result.path, result.contentType, result.contentLength);
}

It resolves to one of three shapes:

Filesystem and stream errors reject instead. cacheAsset unlinks its own temp file first, so a failed download never leaves a partial entry behind.

The options are the middleware's (cacheDir, maxSize, logger, onProgress) plus cacheKey, which the middleware takes from res.locals.cacheKey. The middleware is a thin adapter over this function, so both paths share one implementation and behave identically.

To avoid cluttering your device, an LRU (least recently used) cache eviction strategy is in place. Per default, when your cache dir grows over 1 GB of size, the least recently used (accessed) files will be evicted (deleted), until enough disk space is available again. You can change the cache dir size by specifying options.maxSize (in bytes) when creating the middleware.

$ npm install express-asset-file-cache-middleware

or

$ yarn add express-asset-file-cache-middleware

The URL of the asset to cache.

A unique, expireable cache key. If your asset contains a checksum/digest, you're already done, because it falls back to res.locals.fetchUrl.

To further process the response, the following entries of res.locals are set:

The cached asset as a binary buffer. Most likely, you will end the request chain with

res.end(res.locals.buffer, "binary");

If you're serving your assets in the response, you'll need to set

res.set({
  "Content-Type": res.locals.contentType,
  "Content-Length": res.locals.contentLength
});

You can pass the following options to the middleware:

The root directory where the file cache will be located. Falls back to path.join(process.cwd(), "/tmp").

A logger to use for debugging, e.g. Winston, console, etc.

The maximum size of the cache directory, from which LRU eviction is applied. Defaults to 1 GB (1024 * 1024 * 1024).

A callback invoked repeatedly while an asset is being streamed to disk on a cache miss, useful for driving download-progress UIs. It receives an object { received, total }:

• received - the number of bytes written so far (monotonically increasing, ending at the full size).
• total - the expected size from the upstream Content-Length header, or null when the server doesn't send one (e.g. chunked responses).

fileCacheMiddleware({
  cacheDir: "/tmp",
  onProgress: ({ received, total }) => {
    const pct = total ? Math.round((received / total) * 100) : null;
    console.log(`downloaded ${received} bytes${pct !== null ? ` (${pct}%)` : ""}`);
  }
});

It is only called on a cache miss (an actual download); cache hits are served from disk and emit no progress events.

Run the test suite:

# install dependencies
$ npm install

# unit tests
$ npm test

The MIT License (MIT)

Copyright (c) 2019 Julian Rubisch

Thanks goes to these wonderful people (emoji key):

Julian Rubisch 💻

memalloc 👀

This project follows the all-contributors specification. Contributions of any kind welcome!