V-Legal is being repointed toward a full local-first markdown port of Vietnamese law.

The target product is:

• a filesystem-backed legal corpus where every law document becomes structured markdown
• a rebuildable SQLite index for search, passages, citations, relations, and vectorless RAG
• official-source provenance from vbpl.vn, vanban.chinhphu.vn, and phapdien.moj.gov.vn
• a document-first reading experience inspired by the strengths of thuvienphapluat.vn

This direction is closer to the legalize-es model than to the previous focused-corpus prototype.

This repo currently contains a working FastAPI + SQLite prototype, but the architecture is in transition.

• the old focused economy/finance corpus direction is now legacy
• Appwrite is no longer the target data architecture
• the next major implementation step is a canonical markdown corpus under data/corpus/
• the Hugging Face dataset remains useful for bootstrap and backfill, but not as the final source of truth

As of the latest local build (data/corpus/manifests/corpus_summary.json):

• 153,926 canonical markdown documents under data/corpus/markdown/ (153,620 issued, 306 Phap dien)
• coverage years 1925–2026
• derived SQLite index at data/corpus/sqlite/vlegal.sqlite

Recent search/reader correctness and performance work is tracked in docs/IMPROVEMENT_PLAN.md, including a one-time document_number_norm backfill migration (scripts/migrate_add_search_indexes.py).

flowchart LR
    U[Browser] --> A[FastAPI + Jinja App]
    A --> S[(SQLite Derived Index)]
    S --> M[Local Markdown Corpus]
    V[VBPL] --> R[Raw Source Snapshots]
    C[vanban.chinhphu.vn] --> R
    P[Phap dien] --> R
    H[HF Bootstrap Dataset] --> R
    R --> M

• markdown is the source of truth
• SQLite is derived and rebuildable
• raw source snapshots are preserved for provenance
• official sources outrank third-party snapshots
• deterministic parsing comes before AI-assisted repair
• vectorless RAG should work directly from markdown-derived indexes

• vbpl.vn: primary source for promulgated-document metadata, legal status, relationships, and HTML text when available
• vanban.chinhphu.vn: secondary freshness source for newly issued government texts and official PDFs
• phapdien.moj.gov.vn: primary source for codified topic and effective-law structure
• th1nhng0/vietnamese-legal-documents: bootstrap and backfill source only
• VN-Law-Advisor: reference implementation for crawler ideas only
• thuvienphapluat.vn: UI and workflow benchmark only, not a source dependency

These scripts reflect the current prototype state and will be superseded gradually by markdown-first exporters and rebuilders.

• Python 3.12+
• uv

uv sync
uv run uvicorn vlegal_prototype.app:app --reload --app-dir src

Open http://127.0.0.1:8000.

uv run python -m compileall src scripts
curl http://127.0.0.1:8000/health

These commands cover both the new markdown-first workflow and the older SQLite-first prototype.

# Export canonical markdown files under data/corpus/
uv run python scripts/export_hf_markdown_corpus.py --limit 500 --reset

# Rebuild SQLite from the canonical markdown corpus
set VLEGAL_DATABASE_PATH=data/corpus/sqlite/vlegal.sqlite
uv run python scripts/rebuild_sqlite_from_markdown.py --reset

uv run python scripts/snapshot_official_source.py \
  --source-system vbpl \
  --source-document-id 213327 \
  --url https://vbpl.vn/pages/portal.aspx

# Import one de muc for testing
uv run python scripts/import_phapdien_corpus.py \
  --limit 1 \
  --topic-id 0cf69ad9-6f29-4ee4-8e16-2c3aa65c3a52

# Import the full ebook bundle into data/corpus/markdown/phapdien/
uv run python scripts/import_phapdien_corpus.py

uv run python scripts/backfill_vbpl_references.py

This scans imported Phap dien topics for embedded official vbpl.vn links, builds a local VBPL reference index, and writes official_source_* fields into matching issued markdown documents when the document number matches exactly.

# Import the first law/phap lenh row from the Government Portal listing
uv run python scripts/import_vncp_listing.py --typegroupid 3 --limit 1

# Import the first 10 Government Portal decrees
uv run python scripts/import_vncp_listing.py --typegroupid 4 --limit 10

This uses the server-rendered he-thong-van-ban listing pages, fetches each detail page, emits provisional markdown records, and backfills official_source_* onto matching issued markdown files when no better source is already attached.

# Discover which canonical documents still need PDF OCR
uv run python scripts/build_pdf_ocr_queue.py

# Preview how OCR text maps into the markdown schema
uv run python scripts/preview_pdf_ocr_mapping.py --text-file data/sample_ocr.txt --title "Luật mẫu"

# Apply OCR text back into a canonical markdown document
uv run python scripts/apply_pdf_ocr_text.py \
  --canonical-id vn-issued-vpcp-216529 \
  --text-file data/ocr_output.txt \
  --backend paddleocr_ppstructure_plus_vietocr

# Run the hybrid PDF processor directly
uv run python scripts/process_pdf_attachments.py --canonical-id vn-issued-vpcp-216529

# Run MiniMax text-only OCR post-correction after PaddleOCR
uv run python scripts/apply_minimax_ocr_correction.py --canonical-id vn-issued-vpcp-216529

The main app environment stays on Python 3.14, but the PaddleOCR execution path runs in a dedicated Python 3.13 environment at .venv-ocr/ because paddlepaddle does not currently ship wheels for cp314 on Windows.

MiniMax does not need image support in this architecture. Images and PDFs are handled by PaddleOCR first, then MiniMax is used only on extracted text for post-correction when the OCR sidecar flags a document as optional or recommended for LLM help.

uv run python scripts/bootstrap_hf_dataset.py --limit 500 --reset
uv run python scripts/bootstrap_phapdien_taxonomy.py --seed-only

set VLEGAL_DATABASE_PATH=data/full_hf.sqlite
uv run python scripts/bootstrap_hf_dataset.py
uv run python scripts/bootstrap_relationship_graph.py
uv run python scripts/bootstrap_citation_index.py

uv run python scripts/update_hf_monthly.py --mode full

These commands are transitional. The target pipeline is markdown-first and is documented in docs/FULL_MARKDOWN_PORT_PLAN.md.

Legacy transitional settings still exist in code for Appwrite, but Appwrite is not part of the target architecture.

• Never commit secrets. Real credentials (API keys, tokens, SSH connection strings, passwords) must live only in a local .env file, which is gitignored. Copy .env.example to .env and fill in real values locally.
• .env.example must contain placeholders only (e.g. your_api_key); it is the only env file tracked in git.
• All secrets are loaded from environment variables via src/vlegal_prototype/settings.py. Do not hardcode keys in source, scripts, or docs.
• If a secret is ever committed, treat it as compromised: rotate it immediately at the provider (Appwrite, Netlify, MiniMax, OCI, etc.) and then purge it from git history (git filter-repo/BFG) before force-pushing.
• Before committing, scan staged changes for accidental secrets (e.g. git diff --cached plus a grep for api_key|token|secret|password).

• /
• /tracking
• /documents/{id}
• /compare/{left_id}/{right_id}
• /health

This is still a prototype and is not an authoritative legal-status engine.

Current limits:

• the current import path is still SQLite-first
• Hugging Face remains a bootstrap source, not final source of truth
• citation and lifecycle graphs depend on local corpus quality
• official-source reconciliation is not fully implemented yet
• grounded briefs remain retrieval-based summaries that require official verification

• docs/FOCUSED_CORPUS_SCOPE.md is now a legacy direction note
• docs/DEPLOYMENT_NETLIFY_APPWRITE.md is an archived deployment path
• remaining Appwrite code should be treated as transitional, not strategic

• Design direction: DESIGN.md
• Agent guidance: AGENTS.md
• Development journal: docs/JOURNAL.md
• Memory bank: .agents/memory-bank/