Skip to content

test: bring the 82 unmarked test files into the offline gate (fix rotted path-prefix + SVG tests) - #4035

Open
kokhlo wants to merge 1 commit into
HKUDS:mainfrom
kokhlo:fix/4033-offline-test-gate
Open

kokhlo wants to merge 1 commit into
HKUDS:mainfrom
kokhlo:fix/4033-offline-test-gate

Conversation

@kokhlo

@kokhlo kokhlo commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

CI's offline gate (pytest tests/ -m offline) never ran the 91 files the issue found unmarked — on c7e7a24f it collected 7,743 of 9,428 tests. Of those, 82 files (1,505 tests) carried no marker at all, so no CI job selected them in any mode; pytest_collection_modifyitems only auto-skips integration, so unmarked files are simply invisible.

Every one of the 82 files runs hermetically against mocks / TestClient / monkeypatching (the postgres_impl, qdrant_impl, milvus_impl, neo4j_impl suites construct fakes or patch the connection layer; the api/ suites drive create_app() with LightRAG mocked). They are marked offline with the repo's existing module-level pytestmark = pytest.mark.offline convention. None needed integration/requires_db — the empirical run of the full unmarked set passes with only conditional skips.

Four rotted tests had real causes, all traced to 2ea2814 (2026-08-29) landing after the file was written (08-19) — with CI never running the file, nothing caught the drift:

  • 307-instead-of-200 pair (test_mount_strip_mode_matches, test_mount_verbatim_mode_matches): these exercise the root_path normalization middleware through the WebUI Mount. But create_app only mounts the WebUI StaticFiles when the bun build artifacts exist; on any bare checkout/CI they don't, so /webui/ fell through to the redirect-to-/docs fallback (307) without ever reaching the middleware under test. Fixed by staging a minimal webui/index.html build dir + patching check_frontend_build, the same pattern test_workspace_entry_mount.py already uses.
  • 403-instead-of-401 pair (test_destructive_route_requires_auth_under_a_colliding_prefix[verbatim|strip]): combined_dependency has answered 403 "API Key required" for "API key configured, none presented" since 2ea2814. The property the test is named for is "requires auth", which holds either way; the assertion is now in (401, 403) with the reasoning in the docstring. Which single code should be canonical is left as the maintainer's call, per the issue.
  • 5 SVG tests (surfaced by bringing tests/parser/markdown/ into the gate): cairosvg is a cffi binding — the wheel installs but rendering needs native libcairo, absent from stock macOS and stock python:3.12-slim. The parser itself treats this as an optional capability (check_svg_rasterizer() startup probe). The 4 tests that assert on real PNG output now skip on that probe (mirroring the requires_spacy_models skip pattern), and the cache-hit guard test patches md_parser._rasterize_svg instead of importing cairosvg, so it runs everywhere.

Verification

Before (c7e7a24): uv sync --frozen --extra api --extra offline-storage --extra offline-llm --extra pytest

pytest tests/ -m offline --collect-only -q   → 7,743/9,428 collected (1,685 deselected)
pytest tests/ -m offline                     → 314 failed, 7,382 passed, 48 skipped

After:

pytest tests/ -m offline --collect-only -q   → 9,248/9,428 collected (180 deselected)   (+1,505)
pytest tests/ -m offline                     → 314 failed, 8,766 passed, 169 skipped

The 314 failures are byte-identical on main and on this branch (tests/setup/* — they shell out to scripts/setup/setup.sh, which hard-requires Bash ≥ 4; macOS dev machines ship 3.2). Zero new failures; +1,384 passing; +121 skips are the new conditional guards (spaCy models, cairo native lib, docling). On CI's ubuntu runners (Bash 5) the setup suite passes, so the offline job is expected to be fully green.

The 4 originally-rotted tests plus the SVG set now pass locally (tests/api/test_path_prefixes.py: 34 passed; tests/parser/markdown/: 44 passed + 4 cairo-skips without libcairo, 48/48 with it).

Notes for reviewers

  • Scope: this covers the ~80 unmarked files from the issue (option 1). The remaining deselected 180 are files explicitly marked integration/requires_db/requires_api — not touched.
  • The 401-vs-403 question the issue raises is deliberately not decided here; if you prefer 401 as the canonical "missing credentials" code, that's a one-line change in combined_dependency plus these two assertions going back to == 401.

Closes #4033

CI's offline job (pytest tests/ -m offline) collected 7,743 of 9,428
tests on c7e7a24; 82 files carried no pytest marker at any level, so
no job ever selected them (1,505 tests). All of them run hermetically
against mocks/TestClient -- the full unmarked set passes locally with
only conditional skips (spaCy models, docling, cairo native library).

- mark all 82 files 'offline' via module-level pytestmark, following
  the repo's existing convention (tests/pipeline/ etc.)
- tests/api/test_path_prefixes.py: stage a minimal WebUI build for the
  two Mount tests -- create_app skips the WebUI mount when the bun
  build artifacts are absent, so /webui/ answered 307-to-/docs instead
  of exercising the root_path normalization middleware under test
  (regressed by 2ea2814; the file predates it and was never run by CI)
- tests/api/test_path_prefixes.py: assert the auth challenge on DELETE
  /documents as 'in (401, 403)' -- the property is 'credentials
  required'; combined_dependency has answered 403 for a configured API
  key with none presented since 2ea2814
- tests/parser/markdown/test_parser.py: guard the four real-raster
  SVG tests on check_svg_rasterizer() -- cairosvg is a cffi binding
  and raises without native libcairo, which the parser itself treats
  as an optional capability; mirrors the requires_spacy_models skip
- tests/parser/markdown/test_raw_cache.py: patch
  md_parser._rasterize_svg instead of cairosvg.svg2png in the
  cache-hit guard so the test no longer imports cairosvg at all

Offline selection: 7,743 -> 9,248 collected (+1,505); full run passes.

Closes HKUDS#4033

Signed-off-by: kokhlo <konstantin.khlopkov93@gmail.com>
@kokhlo

kokhlo commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Cross-linking the claim from #4033 (issuecomment-5757542866): PR is up. Local numbers on top of c7e7a24f with exactly the CI sync (uv sync --frozen --extra api --extra offline-storage --extra offline-llm --extra pytest):

  • -m offline --collect-only: 7,743 → 9,248 collected (+1,505; deselected 1,685 → 180)
  • -m offline full run: failures 314 → 314 (byte-identical set — all in tests/setup/*, which shells out to setup.sh requiring Bash ≥ 4 and my dev box is macOS bash 3.2; ubuntu runners have Bash 5), passed 7,382 → 8,766, skips 48 → 169 (new conditional guards: spaCy models, native libcairo, docling)

The one deliberate non-fix: the 401-vs-403 pair now asserts in (401, 403) — the property the test names ("requires auth") is pinned, the canonical code is left to you. Happy to follow up with the == 401 version if that's the preferred contract.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI's offline gate never runs 1,589 tests, and four of them have rotted

1 participant