Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ff5a96a
feat(browser): OPFS-backed BrowserDb mode with close() (#441)
adityamukho Nov 14, 2026
6957e05
test(browser): OPFS mode in a dedicated worker, WAL-less recovery nat…
adityamukho Nov 14, 2026
1f1290a
ci(browser): run OPFS tests in Chrome and Firefox workers (#441)
adityamukho Nov 14, 2026
6729e52
docs(browser): document OPFS lifecycle and recovery guarantees (#441)
adityamukho Nov 14, 2026
fd03fe6
test(browser): record OPFS build sizes and verification results (#441)
adityamukho Nov 14, 2026
291b176
test(browser): publish OPFS browser and memory measurements (#441)
adityamukho Nov 14, 2026
51681ec
fix(browser): validate imports before recovery and complete recovered…
adityamukho Nov 14, 2026
f13abad
test(browser): cover lifecycle and import fault boundaries (#441)
adityamukho Nov 14, 2026
c43845e
fix(browser): check import header access without slicing (#441)
adityamukho Nov 14, 2026
12305fe
docs(browser): align OPFS guarantees and verification evidence (#441)
adityamukho Nov 14, 2026
f3f29b3
test(browser): retain dependents across repeated import recovery (#441)
adityamukho Nov 14, 2026
3d2d5b7
docs(browser): include compiler diagnostics in lint totals (#441)
adityamukho Nov 14, 2026
c3ed047
fix(browser): preserve shared import errors and clean worker helpers …
adityamukho Nov 14, 2026
dd80d40
docs(browser): list every input OPFS import admits in the changelog (…
adityamukho Nov 17, 2026
e689e59
ci: run the browser suites in Safari on macOS (#441)
adityamukho Nov 17, 2026
6813aff
docs: cite WebKit's private-browsing OPFS note (#441)
adityamukho Nov 17, 2026
0331006
Merge branch 'v3' into feat/441-browser-opfs
adityamukho Nov 17, 2026
85f2db5
ci: start safaridriver before the Safari tests (#441)
adityamukho Nov 17, 2026
bb84f39
fix(browser): treat WebKit's InvalidStateError as the OPFS lock (#441)
adityamukho Nov 17, 2026
4708f9b
docs: name Safari's OPFS lock exception under STG-026 (#441)
adityamukho Nov 17, 2026
83589b4
Merge remote-tracking branch 'origin/v3' into feat/441-browser-opfs
adityamukho Nov 17, 2026
4d874f5
docs: record the CI browser results and the Safari lock amendment (#441)
adityamukho Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/wasm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,69 @@ jobs:
- name: wasm-bindgen tests (lib, incl. BrowserDb and IndexedDB)
run: wasm-pack test --headless --${{ matrix.browser }} --features browser

browser-worker:
name: Browser worker tests (headless ${{ matrix.browser }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
browser: [firefox, chrome]
env:
# wasm-bindgen configures the whole lib test binary for a dedicated
# worker; the browser job above retains main-thread coverage.
RUSTFLAGS: --cfg minigraf_browser_worker
# Import interruption sweeps acquire real browser locks at each point.
WASM_BINDGEN_TEST_TIMEOUT: 180
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
- name: Install wasm-pack
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
- name: wasm-bindgen worker tests (OPFS and IndexedDB)
run: wasm-pack test --headless --${{ matrix.browser }} --features browser --lib

safari:
name: Safari tests (${{ matrix.context }})
# Real Safari through safaridriver: Playwright's Linux WebKit has no OPFS
# in workers, so this is the WebKit OPFS lane. Safari has no headless mode;
# the hosted runner's GUI session runs it. Private windows cannot be
# opened over WebDriver, so Safari private browsing is not covered here.
runs-on: macos-15
strategy:
fail-fast: false
matrix:
include:
- { context: main thread, rustflags: "", args: "" }
- { context: dedicated worker, rustflags: "--cfg minigraf_browser_worker", args: "--lib" }
env:
RUSTFLAGS: ${{ matrix.rustflags }}
WASM_BINDGEN_TEST_TIMEOUT: 180
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
- name: Install wasm-pack
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
# The test runner's own safaridriver launch times out on hosted runners
# ("driver failed to bind port"), so start it here and point the runner
# at it with SAFARIDRIVER_REMOTE.
- name: Start safaridriver
run: |
sudo safaridriver --enable
safaridriver -p 4444 &
for _ in $(seq 1 60); do
curl -sf http://127.0.0.1:4444/status >/dev/null && exit 0
sleep 1
done
echo "::error::safaridriver did not start"; exit 1
- name: wasm-bindgen tests in Safari
env:
SAFARIDRIVER_REMOTE: http://127.0.0.1:4444
run: wasm-pack test --headless --safari --features browser ${{ matrix.args }}

wasi:
name: WASI tests (Wasmtime)
runs-on: ubuntu-latest
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **OPFS-backed `BrowserDb` persistence and explicit close** (#441). `await BrowserDb.openOpfs(name)` opens one `<name>.graph` file through an exclusive sync access handle in a dedicated worker on a secure origin. It uses the default 256-page cache, reads pages on demand and commits by copy-on-write with data/meta flushes, without a WAL or a whole-file buffer. IndexedDB `open(name)` now also works in workers. The two modes have independent namespaces and no automatic fallback. OPFS starts at Tier 2 within the Tier 1 browser binding. All modes gain synchronous, idempotent `close()`, which saves nothing, discards an open transaction, invalidates dependents and immediately releases an OPFS lock. OPFS import admits an empty blob, a nonempty v8 image with a selectable meta and complete, verified contents (an empty v8 image included), or a supported v7 image that migrates completely, and rejects anything else (such as a torn or zero-filled initial image) before replacing anything; it validates the whole image, stages and publishes the complete old or new database across interruption, and invalidates earlier cursors/prepared queries after publication. Exports after interrupted initial creation fill an absent second meta slot with zeros and still propagate failed reads of existing pages. OPFS recovery accepts the sole valid meta; unlike native recovery this can silently fall back after later meta rot. A failed commit is unconfirmed (it may be stored), leaves reads working, and refuses later writes with STG-046 until reopen. New registered errors API-026 (unavailable OPFS context), API-027 (busy database) and STG-048 (browser operation and exception name); lock contention is STG-026. No file-format change or new crate dependency. See [durability](docs/DURABILITY.md#browser-storage-modes) and [measurement status](docs/benchmarks/441-opfs/README.md).


- **`OpenOptions::allow_overrides`** (#869, part of #862): the embedder lists, as `Overridable` variants, the query keywords that may override the database's settings. The default, `Overridable::DEFAULTS` (`MaxDerivedFacts`, `MaxResults`), allows every keyword that could do so before, so no query changes. A query using a keyword the list leaves out is refused with the new **API-025** before it runs, from `execute`, `query`, `prepare` and `WriteTransaction::execute`; the list replaces the defaults rather than adding to them, and is not stored in the file. `BrowserDb` takes no options and uses the defaults. `Overridable` is `#[non_exhaustive]`.

- **`:query-memory-limit`, limit bind slots and `PreparedQuery` limit setters** (#870, part of #862). `:query-memory-limit N` replaces `OpenOptions::query_memory_limit` for one query, up or down; `0` is allowed and lets only a query that holds no state run (QRY-012 otherwise). It is gated by the new `Overridable::QueryMemoryLimit`, which is **not** in `Overridable::DEFAULTS`, so a database refuses it with API-025 unless the embedder lists it; `BrowserDb` always refuses it. `:max-derived-facts`, `:max-results` and `:query-memory-limit` take a `$slot` filled with `BindValue::Val(Value::Integer(n))` (INT-014/INT-015 out of range as for a literal, API-024 for another type, API-022 when missing). `PreparedQuery::with_max_derived_facts` and `with_query_memory_limit` replace the database values for one statement; they are not gated, and an allowed keyword still overrides them. See [docs/CONFIGURATION.md](docs/CONFIGURATION.md). The v3 binary-size gate rises from 1,234 to 1,240 KiB (+672 B measured here; the rest is room for #861).
Expand Down
12 changes: 6 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ cargo run < demos/demo_negation.txt
- `dict.rs`: `DictReader` (id ↔ UUID/ident, tx timestamps, long values; `uuids_of` walks dense id batches with `LeafCursor` and uses exact scalar reads for sparse/small batches, #530) and `Encoder` (checkpoint-time id assignment and key building)
- `value_pages.rs`: append-only value pages for strings over 64 bytes and sets whose element encoding is over 64 bytes (#316; DICT dedup compares bytes, `long_bytes`/`find_long_bytes`)
- `reader.rs`: `OnDiskReader` — covering reads of committed facts (`CommittedReader`, `KeySource`); query scans use the resumable `live_keys` to decide net-assert on index keys (#379) among the versions a `TxBound` keeps; `live_facts` is test-only, and production fact-level net-assert handles pending records (#532, superseded by #551); `KeySource::any_key` proves raw kind-prefix presence with one leaf seek, including historical/retracted keys (#641); `tx_order` (cached per reader), `last_tx_at`, `tx_runs` over the DICT's transactions (#551)
- `verify.rs`: integrity walk for `Minigraf::verify` (tree order, index digests, DICT, page accounting) and the source choice for `rebuild_indexes` (#373)
- `verify.rs`: integrity walk for `Minigraf::verify` (tree order, index digests, DICT, page accounting) and the source choice for `rebuild_indexes` (#373); also built with the `browser` feature for OPFS import verification
- `meta.rs` / `page.rs` / `freelist.rs`: meta pages A/B, the common page header and allocator (skips quarantined ids), free-list chain
- `pins.rs`: `Pins` / `GenerationPin` — generations held by open snapshots and the quarantine of pages checkpoints freed after them (#543)
- `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
Expand Down Expand Up @@ -193,11 +193,11 @@ cargo run < demos/demo_negation.txt
- `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation (`storage::pins`), so checkpoints run but do not reuse its pages (`API-013` deprecated, #543)
- `src/log_writer.rs`: `LogWriter` — builds a new file from `FactRecord`s with their tx and valid-time kept (#431): tx-ordered appends with holes (`API-015`/`API-016`), batched copy-on-write checkpoints at tx boundaries, no WAL, built at `<path>.partial` and renamed by `finish` (`STG-043` if the target exists)

7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery; `append_entry` takes the `SyncMode` of each append (#871), and in tests the writer tracks its synced length (lost-sync fault, `Minigraf::power_loss`). Like `storage::verify` and `storage::dir_sync`, not compiled for wasm32 (no file backend there)
7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery; `append_entry` takes the `SyncMode` of each append (#871), and in tests the writer tracks its synced length (lost-sync fault, `Minigraf::power_loss`). Like `storage::dir_sync`, not compiled for wasm32 (no file backend there)

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings
8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings; browser OPFS adds API-026 (unavailable context), API-027 (busy) and STG-048 (failed browser storage operation)

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object; `withMaxDerivedFacts`/`withQueryMemoryLimit`/`withMaxIterations` return a new one, #872; read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `BrowserDb.beginWrite` returns a `BrowserWriteTransaction` (#824: sync `execute` staging through `db::Staged`, `commit()` a `Promise` resolved after the IndexedDB flush, `rollback()`; `BrowserDbInner::tx_open` makes every other read/write call `INT-001`; finished = `API-018`; tests in `tx_tests.rs`); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `backend.rs` (buffer/OPFS dispatch), `opfs.rs` (`OpfsBackend` over an exclusive dedicated-worker sync access handle, one `<name>.graph` file and a 256-page cache; no WAL; sole-valid-meta recovery; crash-atomic import stages `<name>.graph-import`), `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence on the main thread or in workers); `openOpfs(name)` selects OPFS explicitly (Tier 2 in v3.0.0); all modes have synchronous `close()`, invalidating dependents and releasing the OPFS lock, with no save; `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object; `withMaxDerivedFacts`/`withQueryMemoryLimit`/`withMaxIterations` return a new one, #872; read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `BrowserDb.beginWrite` returns a `BrowserWriteTransaction` (#824: sync `execute` staging through `db::Staged`, `commit()` a `Promise` resolved after the IndexedDB flush, `rollback()`; `BrowserDbInner::tx_open` makes every other read/write call `INT-001`; finished = `API-018`; tests in `tx_tests.rs`); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`

### Data Model

Expand Down Expand Up @@ -250,7 +250,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**2159 tests** (2138 active, 21 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**2172 tests** (2149 active, 23 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.
Expand All @@ -273,7 +273,7 @@ Current plan (see `ROADMAP.md` and the milestones):
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **Input order (#641)** targets v3.0.0 in three slices: committed scalar Sort omission, one-group count/count-distinct and partition-at-a-time windows are implemented in the dependent PRs for `v3`. The reviewed design, fallback scope and acceptance gates are in `docs/superpowers/specs/2026-10-28-641-input-order-design.md`.
- **Scalar key decoder proofs (#693)** target v3.0.0 before the release cut. The pure decoder refactor preserves key bytes and INT-049 behavior; Kani covers round trips and bounded malformed inputs. The approved design and CI limits are in `docs/superpowers/specs/2026-10-29-693-decoder-proofs.md`.
- **v3.next** (the release after v3.0.0) holds new features (rule persistence #241; OPFS `BrowserDb` #441).
- **v3.next** (the release after v3.0.0) holds rule persistence (#241). OPFS `BrowserDb` (#441) targets v3.0.0, starts at Tier 2, and is implemented on its feature branch with browser evidence still pending.
- Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md).
- Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2.

Expand Down
13 changes: 11 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,15 @@ wasm-bindgen = { version = "0.2", optional = true }
wasm-bindgen-futures = { version = "0.4", optional = true }
js-sys = { version = "0.3", optional = true }
web-sys = { version = "0.3", optional = true, features = [
"DedicatedWorkerGlobalScope",
"DomStringList",
"Event",
"FileSystemDirectoryHandle",
"FileSystemFileHandle",
"FileSystemGetFileOptions",
"FileSystemHandle",
"FileSystemReadWriteOptions",
"FileSystemSyncAccessHandle",
"IdbDatabase",
"IdbFactory",
"IdbIndex",
Expand All @@ -54,6 +61,7 @@ web-sys = { version = "0.3", optional = true, features = [
"IdbRequest",
"IdbTransaction",
"IdbTransactionMode",
"StorageManager",
"Window",
] }

Expand Down Expand Up @@ -141,8 +149,9 @@ inherits = "release"
workspace = true

[workspace.lints.rust]
# Kani proof harnesses (#627): `cargo kani` sets cfg(kani).
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(kani)', 'cfg(minigraf_shuttle)'] }
# Kani proof harnesses (#627): `cargo kani` sets cfg(kani). The browser tests'
# worker build (#441) sets cfg(minigraf_browser_worker).
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(kani)', 'cfg(minigraf_shuttle)', 'cfg(minigraf_browser_worker)'] }

[workspace.lints.clippy]
# Panic prevention — use Result/? or debug_assert! instead
Expand Down
4 changes: 2 additions & 2 deletions PHILOSOPHY.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ SQLite's success comes from a clear philosophy: be a library, not a server. Be s
- Easy to backup: copy one file
- Easy to share: email, USB drive, version control (for small DBs)
- Easy to delete: remove one file
- WASM: Store in browser's IndexedDB as single blob
- WASM: IndexedDB stores page records; OPFS mode (v3.0.0, dedicated worker) stores one portable `.graph` file, with a transient staging file only during import/recovery

**Anti-pattern**: Multiple files, directories, or complex file structures that are hard to manage.

Expand Down Expand Up @@ -267,7 +267,7 @@ Data-integrity bugs stay visible until they are fixed in a published release; se

A binding moves to Tier 1 when real users need it: issues from outside users, or known dependents. The project's own tools count as dependents when they ship to users and rely on the binding: Python is Tier 1 because `temporal_reasoning` stores its memory through it, and browser WASM is Tier 1 because the playground, the graph visualizer and the docs site run on it. A promotion takes effect at a major or minor release, never in a patch. No binding is removed by being Tier 2.

A new storage backend inside a Tier 1 binding starts at Tier 2 until it has shipped in a release; the OPFS backend for browser WASM (#441, v3.next) is the current example. The IndexedDB backend stays Tier 1.
A new storage backend inside a Tier 1 binding starts at Tier 2 until it has shipped in a release; the OPFS backend for browser WASM (#441, v3.0.0) is the current example. The IndexedDB backend stays Tier 1.

| Tier | Platform and filesystem |
|---|---|
Expand Down
Loading
Loading