pytest: collecting rag/flow/tests/ together with test/unit_test/rag/ corrupts deepdoc.parser (cascading PlainParser ImportError) #17985

Summary

Collecting rag/flow/tests/ and test/unit_test/rag/ together in a single pytest invocation corrupts the cached deepdoc.parser package module in sys.modules, which then makes every from deepdoc.parser import X fail with a cascading ImportError: cannot import name 'PlainParser' from 'deepdoc.parser.pdf_parser' (unknown location).

This is not a deepdoc native-build problem and not a bug in any single test file. Each suite collects cleanly on its own.

Reproduction

Each suite alone is green:

python -m pytest rag/flow/tests/ --collect-only -q          # 12 tests, OK
python -m pytest test/unit_test/rag/test_naive_merge.py --collect-only -q   # 23 tests, OK

Combined collection fails:

python -m pytest rag/flow/tests/test_token_chunker_tag_overlap.py test/unit_test/rag/test_naive_merge.py --collect-only -q
# ERROR collecting rag/flow/tests/test_token_chunker_tag_overlap.py
# ImportError: cannot import name 'PlainParser' from 'deepdoc.parser.pdf_parser' (unknown location)

The failure is order-independent (swapping the two args still fails) and survives --import-mode=importlib.

Root cause

PlainParser is only a symptom. At the moment of failure, sys.modules shows:

deepdoc.parser.pdf_parser  cached: __file__=None  __path__=None  hasPlainParser=False
deepdoc.parser.__path__=None          # the deepdoc.parser package module itself is broken/partial
deepdoc.__path__=['/home/.../ragflow/deepdoc']   # correct

deepdoc.parser.pdf_parser has __file__=None and its SourceFileLoader.exec_module is never invoked, so the real deepdoc/parser/pdf_parser.py is never executed — it was cached as a namespace/partial module. Any later from deepdoc.parser import ... then fails against this poisoned cache, and the error chain cascades into docs_generator, invoke, rag.flow.*, etc.

The trigger is structural to the test layout:

• test/unit_test/rag/ and rag/flow/tests/ are not packages (no __init__.py). pytest's default prepend import mode puts each of these directories on sys.path[0].
• test/unit_test/rag is itself named rag, and test/unit_test/ contains a deepdoc/ directory (test/unit_test/deepdoc/parser/...).
• When both suites are imported in one process — the flow suite via rag.flow.parser.pdf_chunk_metadata → api.db.services → deepdoc.parser, and the test/unit_test suite via rag.nlp and its conftest — deepdoc / deepdoc.parser resolution hits a namespace / partial-module cache collision and deepdoc.parser ends up cached as a non-package broken module.

Note: CPython 3.13 executes most of the import machinery in C, bypassing builtins.__import__, module_from_spec, and SourceFileLoader.exec_module, so the precise first corruption event cannot be observed via Python-level import hooks; the sys.modules end-state above plus the failure conditions are sufficient to localize the mechanism.

Independent verification that deepdoc is fine

python -c "import deepdoc.parser.pdf_parser as m; print(m.__file__)"   # OK
python -c "from api.db.services.task_service import TaskService; print('ok')"  # OK
# pure-python: import rag.nlp then import rag.flow.chunker.token_chunker  -> OK

Suggested fixes (out of scope for this issue)

1. Decouple the import path: rag/flow/parser/pdf_chunk_metadata.py does from api.db.services.file_service import FileService at module top level, which pulls in the entire api.db stack (and thus deepdoc.parser) whenever the chunker package is imported. Moving that heavy import behind a function / lazy import removes the chunker from the deepdoc import chain.
2. Make test directories real packages (add __init__.py to test/unit_test/rag, rag/flow/tests, etc.) so pytest resolves them unambiguously instead of as sys.path roots / namespace portions.
3. Keep the two suites in separate pytest invocations in CI as a stopgap (each already collects green alone).

Workaround for the affected test

rag/flow/tests/test_token_chunker_tag_overlap.py collects and passes cleanly on its own (and will in CI once deepdoc is built). It only breaks when collected in the same process as test/unit_test/rag/. No code change is required in that test.