Self-hosted server flavour: deploy artifact must ship bundle externals + worker sidecar files #5429

Context

The self-hosted server flavour is meant to deploy as a copyable folder — user copies the build output to a server and runs it (a packaging CLI is planned to produce this folder). This works in webiny watch today, but the produced build/ is not self-contained for a real deploy. Two related gaps, both stemming from how the api handler is bundled (packages/build-tools/bundling/function/createRsbuildConfig.js, target: node, everything bundled into build/handler.mjs).

Problem A — runtime sidecar worker files are not shipped

Two server packages load a separate file at runtime via import.meta.url:

• @webiny/background-tasks-server → dist/worker/workerEntry.js (spawned via worker_threads).
• @webiny/api-scheduler-server → dist/jobs/pollWorker.js (handed to bree).

When bundled, rspack:

1. freezes import.meta.url to the build-machine source path, e.g.
   file:///Users/.../packages/background-tasks-server/dist/service/WorkerTaskService.js, and
2. does not emit the sidecar files into build/.

So on a shipped build/, new Worker(...) / bree resolve to a nonexistent path → ENOENT. (In dev it happens to work only because the frozen absolute path points at the real monorepo dist/.)

Confirmed: grep workerEntry build/handler.mjs finds only the string; there is no build/worker/ or build/jobs/, no emitted chunk.

Problem B — bundle externals are not provisioned for the server flavour

The bundler externalizes packages that shouldn't/can't be bundled:

• Native binaries: sharp, better-sqlite3 (via knex) — cannot be turned into JS.
• Environment-provided: aws-sdk.
• Sidecar-file loaders (Problem A): background-tasks-server, api-scheduler-server.

Externals are resolved from node_modules at runtime. Who provides them differs per flavour:

On AWS, externals are ambient (layer + runtime), so build/ alone deploys — leave AWS as-is. On the server, nothing is ambient: every external must be physically included in the deploy artifact. sharp is the loudest example — image processing silently breaks on a server deploy without it, exactly like the worker files.

So the server "self-contained bundle" is really:

build/
├── handler.mjs                 # the bundle
└── node_modules/               # the externals, incl. native binaries + sidecar files
    ├── sharp/
    ├── better-sqlite3/  knex/
    ├── @webiny/background-tasks-server/   # carries dist/worker/workerEntry.js
    └── @webiny/api-scheduler-server/      # carries dist/jobs/pollWorker.js

Problem C — native deps are OS/arch-specific → package in CI

sharp and better-sqlite3 ship per-OS/arch binaries. Assembling the externals node_modules on a Mac bundles the darwin binary → fails on a Linux server. The bundle (handler.mjs) is platform-agnostic JS, but the externals node_modules must be resolved for the target OS/arch — i.e. the packaging step runs in CI on the target platform, or cross-installs (npm --os/--cpu, optionalDependencies).

Proposed direction

1. Externalize @webiny/background-tasks-server + @webiny/api-scheduler-server in createRsbuildConfig.js (mirrors the existing knex rationale). This ships their sidecar files and un-freezes import.meta.url (resolves from node_modules). Safe for AWS (not imported there; no-op like knex).
2. Packaging CLI — new command that produces the shippable folder: build/ + a platform-correct build/node_modules/ containing the full externals set. Runs in CI for the target platform.
3. Deploy artifact = build/ + build/node_modules/ (externals). Document this as the server deploy contract.

Externals the server packaging must provision

sharp, knex, better-sqlite3, @webiny/background-tasks-server, @webiny/api-scheduler-server (+ their transitive deps).

Notes / scope

• Dev (webiny watch) already works and needs no change: build/ sits deep in the monorepo, so Node resolves all externals by walking up to the root node_modules (which has everything, @webiny/* symlinked to packages/*/dist).
• The current PR (feat(self-hosted): background tasks, scheduler & real-time on the server flavour #5414, self-hosted background tasks + scheduler + websockets) intentionally leaves this untouched — watch works there. This issue tracks making the server flavour deployable (externalization + packaging CLI + CI/platform handling), to be done separately.

Related

• PR feat(self-hosted): background tasks, scheduler & real-time on the server flavour #5414 (server-flavour background tasks / scheduler / websockets)