Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

feat: implement multithreaded WASM support, WebGPU fallback, and asse… - #108

Open
Abdodiab2005 wants to merge 2 commits into
supertone-oss-archive:mainfrom
Abdodiab2005:fix-wasm-caching
Open

Abdodiab2005 wants to merge 2 commits into
supertone-oss-archive:mainfrom
Abdodiab2005:fix-wasm-caching

Conversation

@Abdodiab2005

Copy link
Copy Markdown

Summary

The web demo took up to 10 minutes to become usable because ~400 MB of ONNX
weights were being downloaded inefficiently. This PR rewires the dev server,
loader, and ONNX Runtime config so the first load is several times faster and
subsequent loads are near-instant.

What was wrong

  • Sequential downloads. The 4 ONNX models (~400 MB total) were fetched one
    after another in a for loop.
  • Double-download on WebGPU fallback. If WebGPU init failed, the code
    called the loader a second time with executionProviders: ['wasm'],
    re-downloading every model — ~800 MB worst case.
  • No browser caching. The dev-server middleware piped files with no
    Content-Type, Content-Length, ETag, or Cache-Control, so every
    refresh re-fetched the full 400 MB and the UI could not show real progress.
  • WASM threads disabled. No COOP/COEP headers meant SharedArrayBuffer
    was unavailable, so onnxruntime-web ran single-threaded.

Changes

web/helper.js

  • loadOnnx now fetches the model itself as an ArrayBuffer (with streaming
    byte-level progress and cache: 'force-cache') and passes the buffer to
    InferenceSession.create. ORT can switch execution providers without
    re-downloading the weights.
  • loadTextToSpeech downloads all 4 models in parallel via Promise.all
    instead of sequentially.
  • Progress callback now reports aggregated MB downloaded / total MB.

web/main.js

  • Detect WebGPU once with navigator.gpu.requestAdapter(), then call the
    loader once with executionProviders: ['webgpu', 'wasm']. ORT falls
    back per-op internally — no second download.
  • Configure ort.env.wasm up front: SIMD on, numThreads set from
    navigator.hardwareConcurrency (capped at 4) when the page is
    cross-origin isolated.

web/vite.config.js

  • Static-asset middleware now sets Content-Type, Content-Length, ETag,
    Accept-Ranges, and Cache-Control: public, max-age=31536000, immutable,
    with 304 Not Modified short-circuit and HTTP Range support.
  • New COOP/COEP plugin sets Cross-Origin-Opener-Policy: same-origin and
    Cross-Origin-Embedder-Policy: require-corp on all dev/preview responses,
    enabling SharedArrayBuffer (required for multi-threaded WASM).

Test plan

  • npm run dev — first load: 4 .onnx requests fire in parallel in the
    Network tab; status shows live MB progress.
  • Refresh the page — all 4 models return 304 Not Modified (or come
    from the disk cache); UI is ready almost immediately.
  • On a machine with WebGPU: backend badge reads WebGPU, no second
    download in the Network tab.
  • On a machine without WebGPU: falls back to WASM with a single download.
  • In the console: self.crossOriginIsolated === true and
    ort.env.wasm.numThreads > 1.
  • Generate speech end-to-end and verify the audio plays.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant