llm-corpus-pipeline

국문 요약. 로컬 LLM 으로 대규모 텍스트 코퍼스를 분류하고 검색하는 파이프라인입니다. 원형은 6만 건 규모의 문서를 수집, 분류, 색인한 연구 인프라이며, 이 레포는 그중 범용 부분을 도메인 중립적으로 재작성한 것입니다. 구성은 여섯 부분입니다. 파일 기반 작업 큐 데몬, vLLM 기반 2단계 분류 골격, GPU 서빙 진단 노트 3건, FAISS 하이브리드 검색 모듈, 이들 앞에 놓이는 FastAPI 게이트웨이, 그리고 그 게이트웨이 위에 올라가는 에이전트 층입니다. 에이전트 층은 프레임워크 중립 tool 정의, MCP 서버 어댑터, 그리고 검색 결과를 복수로 판정해 판정이 갈린 항목을 다수결로 덮지 않고 검토 큐로 보내는 워크플로로 이루어집니다. 에이전트 층의 데모 시나리오는 공개 논문 색인을 대상으로 한 문헌 triage 이며, 초록 수준에서 판정합니다. 합성 샘플 데이터와 공개 논문 색인 응답 1건을 동봉하므로 클론 직후 네트워크 없이 전체 흐름을 실행해 볼 수 있습니다.

A pipeline for classifying and searching large text corpora with locally served LLMs.

The original system was built as research infrastructure for corpus construction. It processed 62,401 crawled files into 60,777 deduplicated documents and classified them in two phases on self-hosted vLLM. This repository is a domain neutral rewrite of the reusable parts. The sample documents bundled here are synthetic, and the original corpus and the domain specific research prompts are not included. The one exception is samples/literature/, which holds captured responses from a public paper index so the agent layer's demo runs offline. See Notes and limits.

The three diagnosis notes cover GPU idle time caused by oversized max_tokens, throughput tuning with --max-num-seqs, and an FP8 KV cache OOM. Figures quoted in them come from the original deployment logs.

git clone https://github.com/oudeis01/llm-corpus-pipeline
cd llm-corpus-pipeline
python scripts/run_sample_pipeline.py

There is no install step. The core pipeline runs on the Python standard library alone, on Python 3.10 or newer.

The sample run walks the bundled synthetic documents through queueing, two-phase classification, and index build, then issues a hybrid search query. Classification calls any OpenAI compatible endpoint. A small mock backend is included so the sample runs on machines without a GPU. See docs/serving.md for running the real thing on vLLM.

pyproject.toml declares no required dependencies and six optional extras. Install the ones you need:

pip install -e ".[repair]"   # json-repair, the output formatting guard for classification and judging
pip install -e ".[search]"   # sentence-transformers and faiss-cpu, for real dense retrieval
pip install -e ".[queue]"    # psutil, lets the daemon adopt a surviving worker process
pip install -e ".[api]"      # fastapi and uvicorn, for the HTTP gateway
pip install -e ".[mcp]"      # the MCP SDK, for the agent tool server
pip install -e ".[graph]"    # langgraph, for the agent workflow
pip install -e ".[all]"      # all six

The HTTP gateway and the agent layer are the only parts that need installed dependencies. Everything else, including the sample run above, works without them. mcp and graph are cut apart rather than fused into one agent extra so that each adapter installs only what it imports, which makes the layering claim checkable instead of asserted (D09).

pip install -e ".[api]"
scripts/serve_api.sh         # 127.0.0.1:8080, interactive docs at /docs

Nothing here is importable as a package. The install carries dependencies only, and the runners stay executable by path, which is how the queue daemon invokes them. The API keeps that property: it is served with uvicorn app:app --app-dir src/api and reaches the pipeline modules by file path. docs/api.md explains why the ordinary alternative breaks.

The original deployment ran inside a conda environment, managed as one shared interpreter for the whole project rather than one environment per script. Queue task files therefore name their interpreter by absolute path, which is what let a systemd unit reach it instead of falling back to the system Python. Any environment manager works here. The paths in samples/tasks/ are placeholders for your own.

src/agent/ puts an agent surface over the HTTP gateway. It is cut into three layers, and the cut is the claim being made: each layer runs, and installs, without the ones above it.

Layer 1 holds every tool as plain data: the name, a JSON Schema for the input, the description, and the error contract. It imports no agent SDK, so both adapters register the same definition and there is no second copy to drift (D02). The adapters add a protocol and a control flow. Neither adds a tool.

Eleven tools are published. Seven read the pipeline, covering search, index state, task listing, task detail, task logs, queue state, and recent events. One asks the server to reload its index after a rebuild. Two work the review queue, recording a disagreement and listing what is waiting. One submits a pipeline task, and it is the only tool that puts work into the queue. Queue control is deliberately absent, so the agent can create work and can only observe work it did not create (D01). The two directory arguments a submission carries resolve against a workspace root, and anything landing outside it is refused (D13).

Tools reach the pipeline over the same HTTP contract an external client would call, rather than by importing the pipeline modules (D03). The API server has to be running for the agent layer to do anything. The review queue is the one documented exception: nothing in the pipeline serves it, so the agent layer owns the file. If a second consumer ever appeared, it would move behind an endpoint like everything else.

pip install -e ".[api,mcp]"
scripts/serve_api.sh &                                   # the tools call this

python3 src/agent/mcp_server.py --workspace-root <a run directory>

The transport is stdio (D04), so an MCP client launches that command as a child process instead of attaching to a port. The tool surface then holds exactly that client's lifetime and privileges, and no second network door opens beside the API. --base-url, --workspace-root and the rest also read from LCP_API_URL, LCP_WORKSPACE_ROOT and LCP_ESCALATION_DIR, which is what an MCP client configuration file usually has room for.

What the server publishes is the neutral schema, byte for byte. The SDK's high level server derives a schema from the handler's Python signature instead, which drops the enum on phase, the shape of a verdict, every property description, and additionalProperties: false. The low level server publishes the document it is handed, so what a model reads is what tools.py wrote (D14).

To exercise it without an MCP client:

python3 scripts/agent_mcp_probe.py --workspace-root <a run directory>

The probe lists the surface, calls tools, provokes each error code, and round trips the review queue over a real stdio child process. docs/agent-mcp-run.md holds a captured run (D10).

The workflow runs the whole loop: search, judge each hit several times, route disagreements to review, submit pipeline work when coverage is short, wait for the daemon, reload the index, and search again (D05). The transitions are fixed edges. The model is confined to judging one document at a time and never picks the next step, so the routing rule is structure rather than an outcome that happened to occur (D12).

The routing rule is the keynote. Any difference in the verdict label sends the item to the review queue with every judgment kept, and agreement means unanimity (D07). An item waiting in review does not count toward coverage, so the loop cannot declare success on the queries its judges are least sure about. Every run ends with a named reason: coverage_met, no_progress, iteration_cap, wait_timeout, or no_work_available (D08).

pip install -e ".[api,graph]"

scripts/serve_api.sh &                                   # 1. the API
python3 src/queue/daemon.py $WORK/queue &                # 2. the daemon

python3 scripts/agent_graph_demo.py \
    --query "compost pile weeks damp sponge" \
    --workspace-root $WORK \
    --work-input corpus_full --work-output index

Judgments come from a deterministic stub by default, so this needs no GPU and the tests stay reproducible. --judge model --endpoint <url> --model <name> swaps in a served model through the same interface (D06). docs/agent-graph-run.md holds a captured run with the corpus split that gives the loop something to do.

Two served models were measured against the judgment schema, since D06 rests on a claim about what real models do with it. Qwen3-14B-AWQ and Qwen2.5-14B-Instruct-AWQ returned 360 of 360 replies as strict JSON with a verdict from the closed set, so the client side recovery ladder was never reached. docs/judge-model-comparison.md has the method, the two conditions and what the numbers do not settle, and scripts/judge_model_compare.py reruns it.

The workflow's scenario is literature triage: name a topic or an author, gather candidate papers from a public paper index, judge each one for relevance several times, send the ones the judges disagree about to a person, classify what survives, rebuild the index, and search again to check coverage (D11). It generalises a task the pipeline underneath was built for, so the correspondence is close to one to one rather than bent to fit.

Triage runs on abstracts. The paper index serves them directly in its Atom feed, and full text would need PDF extraction, which this repository does not have.

python3 scripts/fetch_literature.py \
    --topic "carbon sequestration" --output-dir $WORK/papers

That writes ordinary pipeline documents, so both classification phases, the index builder, and the workflow read them without knowing where they came from. The subject is required and never defaulted, because running relevance judgments over a named person's work is a reasonable thing to do privately and a discourteous thing to ship as a default in a public repository (D11). It reads a captured feed by default and needs no network; --live queries the index, and --author searches the author list instead of the abstract. --live --save-capture writes the response back as the offline fixture for that subject, so a capture is a command rather than a manual step, and it refuses to replace an existing one unless told to. Where the fetcher lives and what offline means are D15. docs/literature-triage-run.md holds two captured runs over the fifteen abstracts of the first captured subject.

Every fork with more than one defensible answer is recorded under docs/decisions/, one file per decision, written before the code it settles. Each record names the options that were rejected and what would reverse it.

• Reference labels used to evaluate the original classifier were generated by an AI agent, not by human annotators. Treat them as a consistency reference, not a human verified standard.

• The original research corpus and its domain specific prompts stay out of this repository. The original filtering results cannot be regenerated from this code alone, and the code makes no claim that they can.

• Synthetic sample documents are generated text with no relation to the original sources.

• samples/literature/ is the exception to that. It holds captured arXiv API responses for three subjects, each held byte for byte with the request URL and the capture date recorded beside it (D15). The titles, abstracts, and author names in them are real. arXiv places descriptive metadata, which its API terms define as including titles, abstracts, and authors, under a CC0 1.0 public domain dedication, so redistributing them here is permitted; the restriction arXiv does place is on serving e-print content itself, and this repository stops at the abstract. They are bundled so the demo runs without a network, not as material this repository has any claim over. Regenerate any of them with scripts/fetch_literature.py --live --save-capture at any time.

  Thank you to arXiv for use of its open access interoperability. This repository was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.

• The RAG layer's default backend is a deterministic extractive stub. It demonstrates the loop and its citation contract, not generation quality, and no live backend run is recorded here yet.

• Several judgments produced on one GPU with one model loaded are repeated sampling with varied prompts, not a panel of independent judges. The variance is genuine and the routing built on it is real, but it is narrower than independent models would give, and the measurement says how much narrower: across both served models the panel split only along the strict rubric, while balanced and lenient agreed on all 60 items in one model and all but one in the other. Two of the three rubrics are close to the same judge (docs/judge-model-comparison.md).

src/
  queue/       daemon, CLI, task schema
  classify/    phase 1 and phase 2 runners, prompt templates
  search/      FAISS index build, keyword index, query interface
  api/         HTTP gateway: routers, gateways to the queue and the engine
  agent/       tool definitions, API client, MCP adapter, workflow, judges
  rag/         context assembly from classifier results, cited generation
  ingest/      paper index client that turns a subject into pipeline documents
scripts/       serving scripts, sample pipeline entry point, API launcher,
               agent probe, workflow demo, literature fetcher
docs/          serving guide, diagnosis notes, API contract, agent run records
  decisions/   one file per design decision, options rejected included
samples/       synthetic input documents, and a task file template
  literature/  captured paper index responses, so the demo runs offline
tests/         API, agent layer, and ingest test suites

MIT. See LICENSE for the scope note on what the license does and does not cover.