Preparation, structuring and vector indexing of regulatory documents with semantic search.

English · Русский

The application ingests a regulatory document (.docx), reconstructs its logical structure with a large language model, splits it into meaningful fragments, vectorizes them and stores them in Qdrant with rich metadata (structural element type, numbering, hierarchy, document version, tags, links to neighbouring fragments). On top of the index it provides vector search over text and over tables, with filters and context assembly of a configurable width.

The service tracks document versions, rejects exact duplicates, and stores tables as separate entities.

• Parsing of .docx and reconstruction of the document's logical parts (stitching of broken paragraphs, isolation of clauses and sub-clauses).
• LLM structure markup: element type, own numbering, relative depth, amendment flag.
• Building the document hierarchy (sections, clauses, sub-clauses) and flattening it into nodes with links to parent, children and neighbouring fragments in reading order.
• Automatic detection of the document name and version.
• Fragment tagging.
• Extraction and resolution of cross-document/cross-clause references (links to other documents' clauses or to clauses within the same document), with a pending-reference queue that auto-resolves once the referenced document is later ingested.
• Vectorization with an embedding model and ingestion into Qdrant.
• Vector search over texts and over tables with filters (name, version, block, structural type, tags) and a context-width parameter.
• Document listing (aggregated by name + version) with metadata — node count, blocks present, tag union, upload time — and filters (name, version, block, tags, upload-time range).
• Full-text deduplication and versioning with a list of the document's other versions in the store.
• Background processing status stored in Redis.

• Python >=3.11,<3.14.
• uv for dependency management.
• Qdrant (vector database).
• Redis (job statuses and the document/version registry).
• Ollama with two models:
  • an LLM for markup, merging, tagging and version detection;
  • an embedding model (default bge-m3, vector size 1024).
• Docker and Docker Compose — optional, to run the infrastructure and the app in containers.

OCR system libraries (poppler, tesseract) are not required for .docx: parsing goes through partition_docx (python-docx), without heavy backends.

docker compose up -d qdrant redis

Brings up Qdrant (:6333) and Redis (:6379) with data volumes.

The embedding model is mandatory; the LLM is your choice (see docs/en/configuration.md):

ollama pull bge-m3
ollama pull qwen2.5:7b-instruct

Copy the example and adjust addresses and models if needed:

cp .env.example .env

Every setting has a sane default in code (src/common/config/app_config.py), so the app starts without a .env at all; the example covers the service's network links (Ollama, Qdrant, Redis, Kafka addresses) — what is usually overridden. The full variable reference is .env.full.example and docs/en/configuration.md. Variables are prefixed with DVD_. DVD_VECTOR_SIZE must match the embedding model's dimension (bge-m3 = 1024).

uv sync
uv run python -m src.dev_runner

The API docs (Swagger) are available at http://localhost:8000/docs.

The repository ships a Dockerfile and an app service in docker-compose.yaml:

docker compose up -d --build

The addresses of Qdrant, Redis and Ollama for the container are set via environment variables in docker-compose.yaml.

• docs/en/architecture.md — architecture, modules and classes, data model.
• docs/en/pipeline.md — the document processing pipeline stage by stage, deduplication, versioning.
• docs/en/api.md — endpoints, request/response formats, examples.
• docs/en/configuration.md — environment variables and model recommendations.

Russian documentation: README-ru.md and docs/ru/.

The core pipeline is implemented and verified end to end on a real regulatory document (SP 19.13330.2019) with a local Ollama. OCR-free text formats are supported out of the box (.docx, .txt, .md, .html — configurable via DVD_ALLOWED_EXTENSIONS); scanned PDF/OCR is planned.