wiki2rag

Ingests all public pages from a Wiki.js instance into a Qdrant vector store, ready for use in any RAG (Retrieval-Augmented Generation) pipeline. Uses Ollama for embeddings — no cloud API keys required.

Wiki.js GraphQL API
      │
      ▼
  wiki_client.py   ← lists published, non-private pages (+ HTML scraping fallback)
      │
      ▼
   chunker.py      ← structure-aware splitting: headers → tables/lists → word window
      │
      ▼
   embedder.py     ← encodes chunks via Ollama's HTTP API
      │
      ▼
    store.py       ← upserts into Qdrant with payload indexes

Qdrant persists the vectors in a named Docker volume. Any application that can speak to Qdrant's REST or gRPC API can query the same data – no coupling to this ingestion code required.

• Docker and Docker Compose
• Ollama running on the host with an embedding model pulled:

ollama pull nomic-embed-text

cp .env.example .env
$EDITOR .env          # set WIKI_URL and OLLAMA_URL at minimum

Required:

Optional but common:

See .env.example for the full list.

docker compose up -d qdrant

Qdrant's REST API is now available at http://localhost:6333. Browse the built-in dashboard at http://localhost:6333/dashboard.

docker compose build wiki2rag    # only needed once (or after code changes)
docker compose up -d wiki2rag    # runs continuously, polling every hour

The service ingests all pages on startup, then sleeps and re-checks for changes every POLL_INTERVAL_SECONDS (default 3600 = 1 hour). Only pages whose updatedAt timestamp has changed are re-fetched and re-embedded, so repeated runs are fast.

For a one-shot run (ingest once and exit):

POLL_INTERVAL_SECONDS=0 docker compose run --rm wiki2rag

The image is lightweight (~200 MB) since embeddings are handled by Ollama — no ML frameworks or models baked in.

After page ingestion, wiki-level metadata chunks are automatically generated and stored — page count, contributors, tags, recently updated pages, newest/oldest pages, and a full page listing. This lets a chatbot answer questions like "how many pages does the wiki have?" or "who are the top contributors?" without needing a separate data source.

Note: Changing models requires a new Qdrant collection (different dimensions). Set a new QDRANT_COLLECTION name and run with FORCE_REINGEST=true.

query.py lets you search the collection directly from the terminal.

Via Docker (recommended):

# Basic search
docker compose run --rm query "how do I reset my password?"

# More results
docker compose run --rm query --limit 10 "event manager setup"

# Show matching text under each result
docker compose run --rm query --show-text "access control"

# Filter out weak matches
docker compose run --rm query --min-score 0.6 "door code"

# Different collection
docker compose run --rm query --collection wiki-internal "onboarding"

# All options
docker compose run --rm query --help

Locally (outside Docker):

QDRANT_HOST=localhost python src/query.py "how do I reset my password?"

Options:

Any language/framework with a Qdrant client can query the store directly.

Python example:

import httpx
from qdrant_client import QdrantClient

client = QdrantClient(host="localhost", port=6333)

# Get embedding from Ollama. If your Ollama endpoint needs auth, add:
# headers={"Authorization": "Bearer <OLLAMA_API_KEY>"}
resp = httpx.post("http://localhost:11434/api/embed", json={
    "model": "nomic-embed-text",
    "input": ["How do I reset my password?"],
})
query_vector = resp.json()["embeddings"][0]

results = client.query_points(
    collection_name="wiki",
    query=query_vector,
    limit=5,
    with_payload=True,
).points

for hit in results:
    p = hit.payload
    print(f"{hit.score:.3f}  {p['page_title']}  {p['page_url']}")
    # Use p["context"] for LLM grounding – includes title, section, description
    print(p["context"][:300])
    print()

• Use context for LLM grounding, not text. The context field includes the page title, description, and section breadcrumb, giving the LLM enough surrounding information to produce accurate answers.

• Filter by tags for scoped search:

  from qdrant_client.models import Filter, FieldCondition, MatchValue
  results = client.query_points(
      collection_name="wiki",
      query=query_vector,
      query_filter=Filter(must=[
          FieldCondition(key="tags", match=MatchValue(value="safety"))
      ]),
      ...
  )

• Small-to-big retrieval: search returns small, focused chunks. Use parent_chunk_index to fetch all sibling chunks for broader context, or follow prev_chunk_id / next_chunk_id to expand the window.

• Deduplicate results: if multiple pages share boilerplate text, several top-k results may be near-identical. Group results by content_hash and keep only the best-scoring hit per hash before sending context to the LLM.

• Wiki metadata questions: questions like "how many pages?", "who are the contributors?", or "what was recently updated?" are answered by the metadata chunks (where is_meta = "true"). These are embedded alongside regular content so semantic search finds them naturally — no special query filter needed.

The service runs continuously by default — after each ingestion it sleeps for POLL_INTERVAL_SECONDS (default 1 hour) then re-checks for changes. Ingestion is incremental, so only pages whose updatedAt timestamp has changed are re-fetched and re-embedded.

# Change the polling interval (e.g. every 30 minutes)
POLL_INTERVAL_SECONDS=1800 docker compose up -d wiki2rag

FORCE_REINGEST=true POLL_INTERVAL_SECONDS=0 docker compose run --rm wiki2rag

wiki2rag/
├── src/
│   ├── main.py          # ingestion entry point / orchestration
│   ├── query.py         # command-line query tool
│   ├── wiki_client.py   # Wiki.js GraphQL client (+ HTML scraping fallback)
│   ├── chunker.py       # structure-aware text chunking
│   ├── embedder.py      # Ollama embedding backend
│   ├── store.py         # Qdrant wrapper with payload indexes
│   └── version.py       # version from VERSION file
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
├── VERSION
└── .env.example

1. Log in as an administrator and go to Administration → API Access.
2. Create a new key with Read permission on Pages.
3. Copy the token into .env as WIKI_API_KEY.

An API key is only required if your wiki restricts guest access to the GraphQL endpoint. Many public wikis work fine without one.