Skip to content

Add Apple Silicon Metal guardrails#600

Open
luoyuctl wants to merge 1 commit into
tobi:mainfrom
luoyuctl:fix/apple-silicon-metal-guardrails
Open

Add Apple Silicon Metal guardrails#600
luoyuctl wants to merge 1 commit into
tobi:mainfrom
luoyuctl:fix/apple-silicon-metal-guardrails

Conversation

@luoyuctl
Copy link
Copy Markdown

Summary

  • apply Apple Silicon Metal workarounds before node-llama-cpp initializes
  • disable Metal residency sets by default on macOS arm64 to avoid exit-time GGML_ASSERT([rsets->data count] == 0) crashes
  • disable the Metal tensor probe on M5/A19-class chips to avoid shader probe failures before fallback
  • fall back to retrieval order when reranking is unavailable instead of aborting the whole query
  • add targeted tests and document the new opt-out in CLAUDE.md

Why

On Apple Silicon, qmd query currently fails in two different ways:

  1. Processes can abort on exit inside ggml_metal_device_free due to residency set cleanup assertions.
  2. On M5-class devices, the Metal tensor probe can fail during shader compilation even though the rest of the Metal path is still usable.
  3. When rerank context creation fails, qmd query exits non-zero even though retrieval already succeeded.

This patch keeps the GPU path enabled, applies the minimal Metal workarounds before llama init, and degrades reranking gracefully when the reranker backend is unavailable.

Verification

  • pnpm exec vitest run test/llm.test.ts --reporter=verbose -t "Apple Silicon Metal workarounds"
  • pnpm exec vitest run test/llm.test.ts --reporter=verbose -t "QMD_APPLE_METAL_WORKAROUNDS resolution"
  • pnpm exec vitest run test/store.test.ts --reporter=verbose -t "rerank falls back to retrieval order when reranker fails"
  • pnpm exec tsx src/cli/qmd.ts query "test" -n 1

Notes

  • pnpm build currently fails in this checkout with an unrelated existing TypeScript error in src/store.ts (Property transaction does not exist on type Database). I did not modify that code path.
  • QMD_APPLE_METAL_WORKAROUNDS=off opt-outs of the automatic Metal env patching if someone wants the old behavior.

Fixes #368
Fixes #388

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant