Skip to content

Diarization hangs with no error when the pyannote model download stalls #97

Description

@smontplais

Problem

Transcription completes, then the UI parks on Loading speaker diarization model...
and never moves. No error dialog, no error in talktrack.log, no way out but killing
the process. Observed twice on 2026-09-10 (10:05 and 12:22), both times killed
manually after minutes of no progress.

The app log ends mid-download and says nothing useful:

[INFO] httpx: HTTP Request: HEAD .../embedding/pytorch_model.bin "HTTP/1.1 302 Found"
[ERROR] talktrack: embedding/pytorch_model.bin:   0%|   | 0.00/26.6M [00:00<?, ?B/s]

(The [ERROR] level on those lines is just the stderr redirect catching a tqdm bar,
and the symlinks UserWarning nearby is unrelated. Both are red herrings that cost
real debugging time.)

The cause is only visible in the Xet client's own log,
~/.cache/huggingface/xet/logs/xet_*.log:

xet-read-token ... Os { code: 10054, kind: ConnectionReset,
  message: "An existing connection was forcibly closed by the remote host." }
Connection attempt 1/5 ... Connection attempt 2/5 ...

Then it stops logging entirely. hf_xet (1.5.1 here) is a separate Rust client used
for every LFS file, and it does not honour HF_HUB_DOWNLOAD_TIMEOUT. When its
token request gets reset, routine behind a filtering corporate network, the transfer
parks at 0 bytes and nothing propagates to Python. Intermittent, not a blanket block:
another xet-token call in the same second returned 200, and plda.npz downloaded
fine. The 10:05 run died one file earlier, on plda.npz.

Three separate defects turn that network fault into an unrecoverable hang:

  1. diarizer._get_pipeline holds _PIPELINE_CACHE_LOCK across
    Pipeline.from_pretrained
    , which downloads ~33 MB on first use. One stalled
    download therefore wedges the cache for every later attempt too, not just its own.
  2. No timeout and no error path. from_pretrained takes no timeout, and
    DiarizationWorker has no watchdog, so there is nothing to convert a stall into
    the error signal the worker already knows how to emit.
  3. dependency_checker.check_pyannote_models only tests cache_dir.exists().
    With 4 of 5 files present, System Status reported "Speaker diarization model is
    cached."
    The one diagnostic tool in the app actively pointed away from the cause.
    This is what made the problem hard to find, and it is arguably the worst of the three.

State left on disk after a stall: the model directory present but missing
embedding/pytorch_model.bin, plus zero-byte blobs/*.incomplete files and an
orphaned .locks/*.lock.

Scope

  • app/utils/hf_env.py (new): apply HuggingFace download settings, respecting any
    value already set so re-enabling Xet stays possible.
  • main.py: call it before any import that reaches huggingface_hub
    (app.main_window -> transcriber -> faster_whisper -> huggingface_hub). The hub
    freezes HF_HUB_DISABLE_XET into a module constant at import time, so ordering is a
    hard requirement, not a preference. Verified: setting it after import has no effect.
  • app/transcription/diarizer.py: load outside the lock, double-checked, first writer
    wins so two concurrent callers never hold two copies of the same model.
  • app/utils/dependency_checker.py: verify the required files individually and report
    leftover .incomplete blobs.

Out of scope, deliberately: no explicit timeout plumbing. The plain HTTPS path already
applies HF_HUB_DOWNLOAD_TIMEOUT, which defaults to 10s, so leaving Xet off is enough
to make a stall raise. Also out of scope: pre-downloading models from the System Status
panel, which is worth its own issue.

Acceptance criteria

  • HF_HUB_DISABLE_XET is set before huggingface_hub is imported, pinned by a test
    that reads main.py and asserts the call precedes the app.main_window import.
  • An explicit HF_HUB_DISABLE_XET=0 in the environment survives, and applying the
    settings twice is a no-op.
  • _PIPELINE_CACHE_LOCK is provably not held while from_pretrained runs.
  • Two threads loading at once get the same pipeline object back.
  • A load that raises leaves the cache empty and the lock released.
  • check_pyannote_models fails and names the missing file when
    embedding/pytorch_model.bin is absent, and fails when .incomplete blobs remain.
  • A complete cache still passes.
  • Full suite green.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions