This repository was archived by the owner on Sep 9, 2026. It is now read-only.
feat: implement multithreaded WASM support, WebGPU fallback, and asse… - #108
Open
Abdodiab2005 wants to merge 2 commits into
Open
Abdodiab2005 wants to merge 2 commits into
Abdodiab2005 wants to merge 2 commits into
Conversation
…t caching via custom Vite middleware
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The web demo took up to 10 minutes to become usable because ~400 MB of ONNX
weights were being downloaded inefficiently. This PR rewires the dev server,
loader, and ONNX Runtime config so the first load is several times faster and
subsequent loads are near-instant.
What was wrong
after another in a
forloop.called the loader a second time with
executionProviders: ['wasm'],re-downloading every model — ~800 MB worst case.
Content-Type,Content-Length,ETag, orCache-Control, so everyrefresh re-fetched the full 400 MB and the UI could not show real progress.
SharedArrayBufferwas unavailable, so onnxruntime-web ran single-threaded.
Changes
web/helper.jsloadOnnxnow fetches the model itself as anArrayBuffer(with streamingbyte-level progress and
cache: 'force-cache') and passes the buffer toInferenceSession.create. ORT can switch execution providers withoutre-downloading the weights.
loadTextToSpeechdownloads all 4 models in parallel viaPromise.allinstead of sequentially.
web/main.jsnavigator.gpu.requestAdapter(), then call theloader once with
executionProviders: ['webgpu', 'wasm']. ORT fallsback per-op internally — no second download.
ort.env.wasmup front: SIMD on,numThreadsset fromnavigator.hardwareConcurrency(capped at 4) when the page iscross-origin isolated.
web/vite.config.jsContent-Type,Content-Length,ETag,Accept-Ranges, andCache-Control: public, max-age=31536000, immutable,with
304 Not Modifiedshort-circuit and HTTPRangesupport.Cross-Origin-Opener-Policy: same-originandCross-Origin-Embedder-Policy: require-corpon all dev/preview responses,enabling
SharedArrayBuffer(required for multi-threaded WASM).Test plan
npm run dev— first load: 4.onnxrequests fire in parallel in theNetwork tab; status shows live MB progress.
304 Not Modified(or comefrom the disk cache); UI is ready almost immediately.
WebGPU, no seconddownload in the Network tab.
self.crossOriginIsolated === trueandort.env.wasm.numThreads > 1.