Ragloom

A tiny Logstash-like ingestion daemon for RAG.

Point Ragloom at a folder. It watches local files, chunks documents, generates embeddings, and upserts deterministic points into Qdrant.

Use it when you want a small, inspectable ingestion pipeline instead of a full RAG platform.

Why Ragloom?

Most RAG tools are full frameworks or platforms. Ragloom only handles ingestion.

It is built for developers who want to:

• keep a vector database in sync with local documents
• rerun ingestion safely without duplicate chunks
• version chunking strategies explicitly
• inspect what was indexed and why
• avoid adopting a full RAG framework

Status

Ragloom v1.0.0-rc.1 is a release candidate for the first stable operator contract. It is intended for production-like validation before v1.0.0. Report compatibility regressions against the contract below before the final release.

Core v1 support boundary:

• local filesystem source
• polling S3 source
• recursive scanning of regular files under one configured directory
• UTF-8 text, Markdown, source code, text-extractable PDF loading, and deterministic DOCX text extraction
• recursive, Markdown-aware, and code-aware chunking
• OpenAI and generic HTTP embedding APIs
• Qdrant sink
• deterministic point IDs
• persistent local WAL state
• bounded in-process retry for transient ingest failures
• pretty and JSON structured logs
• opt-in local health and metrics endpoint

Feature-gated paths:

• fastembed local semantic signal support when built with --features fastembed

Experimental or best-effort paths:

• semantic chunking remains experimental and opt-in
• fastembed remains a feature-gated semantic provider
• macOS release artifacts remain best-effort convenience binaries rather than release-blocking targets

Not supported yet:

• broad job-management or dead-letter queue subsystems beyond the local failed-work journal
• built-in collection lifecycle management

v1 support matrix

v1 compatibility boundary

The v1 compatibility boundary covers operator-visible point identity, Qdrant payload shape, CLI defaults, durable state upgrades, and supported release targets. v1.0.0-rc.1 freezes this contract for release-candidate validation. The Rust library API remains preview during the RC series; breaking Rust API changes must be documented in a later RC, and the public Rust API becomes SemVer-stable with the final v1.0.0 release.

The Qdrant point ID is the chunk identity. It is derived from the canonical source identity, chunk index, and exact chunker strategy fingerprint. The same inputs keep the same ID. Strategy-fingerprint changes intentionally open a new point-ID space. This prevents changed chunk boundaries or semantics from overwriting older chunks under reused IDs. Operators should reindex and then drop or garbage-collect old points when they want a clean collection. The identity is not duplicated as a chunk_id payload field.

For v1, the stable Qdrant payload fields are canonical_path, doc_id, tenant_id, file_extension, size_bytes, mtime_unix_secs, chunk_index, total_chunks, previous_chunk_id, next_chunk_id, chunk_start_byte, chunk_end_byte, chunk_char_len, chunk_text_sha256, and strategy_fingerprint. Their names, presence, and JSON value kinds are the compatibility contract. chunk_text is optional compatibility data: v1 emits it, but consumers should tolerate its omission. Newly added payload fields are non-contractual until they are explicitly added to this list. Identity, extension, hash, and strategy fields are strings; size, time, index, count, offset, and length fields are non-negative integers; neighboring chunk IDs are either UUID strings or null.

The default embedding backend remains OpenAI with https://api.openai.com/v1/embeddings and text-embedding-3-small; the default chunker mode remains router, using character sizing with max=2000, min=0, and overlap=0. The default state path remains .ragloom/wal.ndjson, collection bootstrap and the health endpoint remain disabled, and the retry defaults remain 3 attempts, 128 queued retries, 100 ms initial backoff, and 2000 ms maximum backoff. Semantic chunking remains experimental and opt-in.

Additive payload fields and bug fixes that preserve these identities, fields, defaults, and readable state are compatible. Removing or renaming a stable payload field, changing a default, reusing an old ID for different chunk content, or making released state unreadable is incompatible. Incompatible changes require release-note migration guidance. See the state-specific rules below for the supported upgrade path.

Quickstart

This example runs Ragloom from source against a local Qdrant instance and the default OpenAI embedding backend.

1. Start Qdrant

docker run -d --name ragloom-qdrant -p 6333:6333 qdrant/qdrant

2. Prepare example documents

mkdir -p docs
printf "Ragloom watches files and indexes chunks into Qdrant.\n" > docs/intro.md

3. Run Ragloom

cargo run --release -- \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --state-path ./.ragloom/wal.ndjson \
  --create-collection-if-missing \
  --openai-api-key "$OPENAI_API_KEY"

To validate startup wiring without ingesting, use one of the non-ingesting command paths first:

cargo run --release -- check \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --create-collection-if-missing \
  --openai-api-key "$OPENAI_API_KEY"

cargo run --release -- dry-run \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --create-collection-if-missing \
  --openai-api-key "$OPENAI_API_KEY"

check validates configuration, source wiring, chunker selection, bootstrap prerequisites, and local durable state without starting ingest. State preflight validates that existing WAL and failed-work journals are readable, use supported record formats, and are writable for a real run. Missing state directories succeed when their nearest existing parent is writable; the check does not create them. check prints the state path and concise journal statuses. dry-run performs the same validation and also prints the effective startup choices, including source kind, chunker selection, and whether Ragloom would bootstrap the configured collection. Neither command creates state, sends embeddings, or writes to Qdrant.

With the default OpenAI model, text-embedding-3-small, Ragloom can infer the Qdrant vector size automatically during bootstrap.

Pass --collection-vector-size <n> when Ragloom cannot infer the size for your embedding backend or model:

• required for --embed-backend http
• required for unknown or custom OpenAI embedding models
• optional if you want to override the inferred size explicitly

Example with an explicit size:

cargo run --release -- \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --create-collection-if-missing \
  --collection-vector-size 1536 \
  --openai-api-key "$OPENAI_API_KEY"

Ragloom only bootstraps the target collection when it is missing. It does not manage broader collection lifecycle tasks such as reconfiguration, deletion, migrations, or index tuning.

4. Expected result

Success looks like this:

• Ragloom starts and keeps running until you stop it with Ctrl+C
• you see startup and ingestion logs instead of a ragloom.fatal error
• you see a structured ragloom.ingest.summary event with counts such as discovered_files, indexed_files, emitted_points, and failed_files
• points appear in the Qdrant collection docs

Installation

Ragloom publishes GitHub Release binaries for supported platforms:

• ragloom-v<version>-x86_64-unknown-linux-gnu.tar.gz
• ragloom-v<version>-aarch64-unknown-linux-gnu.tar.gz
• ragloom-v<version>-x86_64-pc-windows-msvc.zip

Best-effort macOS assets are published with the same naming pattern when those jobs succeed:

• ragloom-v<version>-x86_64-apple-darwin.tar.gz
• ragloom-v<version>-aarch64-apple-darwin.tar.gz

Each release also includes a matching .sha256.txt checksum file for archive verification.

Install a release binary

Download the archive for your platform from the GitHub Release page, extract it, and run ragloom (or ragloom.exe on Windows).

Examples:

tar -xzf ragloom-v1.0.0-rc.1-x86_64-unknown-linux-gnu.tar.gz
./ragloom --version

Expand-Archive .\ragloom-v1.0.0-rc.1-x86_64-pc-windows-msvc.zip -DestinationPath .
.\ragloom.exe --version

If you prefer or need an unsupported target, install from source with Cargo.

Build from source

Building Ragloom requires Rust 1.88 or newer. The latest stable Rust release is recommended.

git clone https://github.com/ragloom/ragloom
cd ragloom
cargo build --release

The compiled binary will be available at:

target/release/ragloom

Verify the built binary version with:

target/release/ragloom --version

Install into Cargo's bin directory

git clone https://github.com/ragloom/ragloom
cd ragloom
cargo install --path .

Then confirm the installed executable with ragloom --version or ragloom -V.

Configuration

Ragloom supports a small typed YAML config for source, embed, and sink wiring.

Basic configuration

source:
  kind: "filesystem"
  root: "./docs"

embed:
  endpoint: "https://api.openai.com/v1/embeddings"

sink:
  qdrant_url: "http://localhost:6333"
  collection: "docs"

state:
  path: ".ragloom/wal.ndjson"

retry:
  max_attempts: 3
  max_queued: 128
  initial_backoff_ms: 100
  max_backoff_ms: 2000

# Optional. Omit to keep the health endpoint disabled.
health:
  addr: "127.0.0.1:8080"

Run with:

ragloom --config ./ragloom.yaml --openai-api-key "$OPENAI_API_KEY"

S3 source configuration

For polling S3 ingestion, use the canonical S3 config shape:

source:
  kind: "s3"
  bucket: "docs-bucket"
  prefix: "kb/"

embed:
  endpoint: "https://api.openai.com/v1/embeddings"

sink:
  qdrant_url: "http://localhost:6333"
  collection: "docs"

Example CLI startup for the same source:

ragloom \
  --source-kind s3 \
  --s3-bucket docs-bucket \
  --s3-prefix kb/ \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --openai-api-key "$OPENAI_API_KEY"

Generic HTTP embedding

For a generic HTTP embedding service:

embed:
  endpoint: "http://localhost:8080/embed"

ragloom --config ./ragloom.yaml --embed-backend http --embed-model default

Configuration notes

• source.kind: filesystem plus source.root is the canonical filesystem config shape
• legacy filesystem config that only sets source.root remains supported for compatibility
• source.kind: s3 plus source.bucket and optional source.prefix configures polling S3 ingestion
• --config can provide source.kind, source.root, source.bucket, source.prefix, embed.endpoint, sink.qdrant_url, and sink.collection
• --config can also provide state.path; the CLI flag is --state-path
• --config can provide retry.max_attempts, retry.max_queued, retry.initial_backoff_ms, and retry.max_backoff_ms
• --config can provide health.addr; the CLI flag is --health-addr
• --dir remains the filesystem CLI shorthand; --source-kind filesystem --dir ./docs is equivalent to filesystem config
• --s3-bucket and --s3-prefix require --source-kind s3
• S3 runtime auth and region come from the process environment; set AWS_REGION or AWS_DEFAULT_REGION plus your normal AWS credential chain inputs
• backend-specific auth still comes from CLI flags, such as --openai-api-key
• chunker settings are currently configured by CLI flags, not by YAML
• check and dry-run are non-ingesting command paths; dry-run reports effective startup choices and bootstrap prerequisites only
• flags support both --flag value and --flag=value
• the config file is merged with CLI flags; CLI flags take precedence
• when --config is in use, Ragloom polls the config file for changes once per second
• hot reload currently applies only to retry.*
• CLI-provided retry flags stay pinned across reloads; file edits only affect retry fields that still come from YAML
• changes to source.*, embed.endpoint, sink.*, state.path, and health.addr are rejected during reload and require a process restart
• invalid or rejected reloads are logged and the last good runtime config remains active

Retry behavior

Ragloom retries transient loader I/O, embedding, and sink failures inside the worker before marking a file version failed for the ingest window. Configuration and invalid-input errors are not retried.

Defaults:

• max_attempts: 3
• max_queued: 128
• initial_backoff_ms: 100
• max_backoff_ms: 2000

CLI overrides:

• --retry-max-attempts <n>
• --retry-max-queued <n>
• --retry-initial-backoff-ms <ms>
• --retry-max-backoff-ms <ms>

Set --retry-max-attempts 1 to disable retries. Backoff is deterministic and jitter-free so tests and local runs remain reproducible.

Source scanning behavior

Ragloom ships with two polling source shapes:

• filesystem: walks the configured root recursively and ingests regular files it can stat
• s3: lists objects under the configured bucket and optional prefix, then ingests them through the same planner/runtime flow

For S3 runtime ingestion, Ragloom treats object keys as opaque. The canonical S3 document identity will be s3://{bucket}/{exact-key} without normalizing duplicate slashes, dot segments, or the configured prefix.

• traversal is deterministic because directory entries are processed in sorted path order
• hidden files and hidden directories are treated like any other path
• symbolic links are not followed
• unreadable directories or files that cannot be stat'ed are skipped

How is Ragloom different?

Ragloom is not a RAG framework, chatbot, document QA app, or observability platform.

It only focuses on ingestion.

Ragloom is for people who already have an app and a vector database, but want a small ingestion process in between.

Core Concepts

Source

Discovers document versions from a location such as a local folder.

Loader

Reads document content. The built-in loaders currently extract UTF-8 text, deterministic PDF text, and deterministic DOCX text from local files and S3 objects behind the same document-loading boundary.

PDF extraction is text-only. Ragloom does not perform OCR, does not reconstruct rich layout, and may return an empty string for image-only or scan-only PDFs. Encrypted or malformed PDFs fail at the loader boundary with project-level errors.

DOCX extraction is also text-only. Ragloom linearizes paragraphs with newline separators and table cells with tab separators, does not preserve rich formatting or embedded assets, and fails malformed DOCX inputs at the loader boundary with project-level errors.

Chunker

Splits documents into indexable chunks and records chunk metadata.

Embedder

Turns chunks into vectors through OpenAI or a generic HTTP embedding API.

Sink

Writes vectors and metadata into a destination such as Qdrant.

State

Tracks discovered work and acknowledgements in an append-only local WAL.

By default, Ragloom stores state at .ragloom/wal.ndjson relative to the current working directory. Pass --state-path <path> or set state.path in the YAML config to choose another file.

The WAL is newline-delimited JSON and records planned WorkItemV2 entries plus SinkAckV2 acknowledgements. On startup, Ragloom replays work items that do not have a matching acknowledgement, seeds planner de-duplication from the WAL so already acknowledged file versions are not planned again, and restores the set of previously observed document paths used by delete detection. This means delete synchronization survives restarts as long as you reuse the same --state-path or state.path. Corrupt or unreadable state files fail startup with a state error instead of being ignored. Ragloom keeps the WAL append handle open for the lifetime of the process to avoid repeated open/close overhead, while still calling sync_data() after each record so the durability boundary remains one acknowledged append at a time.

State compatibility contract

Ragloom v1.0.0-rc.1 directly reads the supported released v0.4.x and v0.5.0 WAL formats, with v0.4.0 as the minimum supported on-disk WAL version. failed.ndjson enters that compatibility surface in v0.4.1; older state directories may not have a failed-work journal yet.

These files are durable user-owned state. Ragloom does not silently skip or repair unknown future record variants, malformed lines, truncated final writes, or other unsupported state shapes. Instead, it fails closed with a state error that identifies the failing file and line so operators can inspect the journal intentionally.

If a future Ragloom release needs an incompatible state change, the project will document that boundary explicitly and require a deliberate migration path rather than silently reinterpreting older state.

Retries are not persisted as separate WAL records. If the process stops while retries are queued, unacknowledged work is replayed from the WAL on the next startup.

When a work item exhausts retries or fails terminally without retry, Ragloom also appends a sanitized failed-work record to failed.ndjson in the same directory as the WAL. Failed-work records include the original scheduled WalRecord, retry attempt count, and a small failure classification only; they do not include secrets, embedding payloads, or full document contents.

Inspect failed work directly from the local NDJSON file, for example:

cat .ragloom/failed.ndjson

To requeue pending failed work back into the WAL for operator replay, run:

ragloom replay-failed --state-path .ragloom/wal.ndjson

Or, if you already keep the state path in YAML:

ragloom replay-failed --config ./ragloom.yaml

replay-failed requires either --state-path or --config. It does not require runtime ingest flags such as --dir, --qdrant-url, or --collection. On success it prints deterministic pending, requeued, skipped, and failed counts. skipped counts exhausted records already marked as requeued. An unsafe or incomplete replay returns a non-zero exit status and includes the partial summary in its error context. Replay is intentionally at-least-once: if a prior replay appended work into the WAL but crashed before marking the failed record requeued, rerunning replay-failed may append that work again rather than silently losing it.

To compact both journals without changing replay semantics, run:

ragloom compact-state --state-path .ragloom/wal.ndjson

Or reuse YAML state configuration:

ragloom compact-state --config ./ragloom.yaml

compact-state is an explicit operator command. Ragloom does not compact state implicitly during normal ingest. The command rewrites wal.ndjson and failed.ndjson to the minimum records needed to preserve startup replay, planner de-duplication, delete synchronization, pending failed work, and replay-failed behavior.

Compaction writes a same-directory temporary file, flushes it with sync_data(), and only then replaces the original journal. On Linux and other Unix targets, Ragloom uses same-directory rename plus parent-directory sync. On Windows, Ragloom uses the native file-replacement primitive so compaction can replace an existing journal without first deleting it. If validation, writing, or replacement fails, the original readable journal remains the safety boundary and Ragloom returns a state error instead of silently discarding records.

Architecture

local folder
  ->
scanner
  ->
planner
  ->
WAL work items
  ->
runtime queue
  ->
loader
  ->
chunker
  ->
embedder
  ->
qdrant sink
  ->
acknowledgement

The implementation is intentionally split into small modules such as source, doc, transform, embed, sink, pipeline, and observability, but the runtime behavior stays narrow: discover files, turn them into chunks, embed them, and upsert them.

Safe Reruns With Deterministic IDs

Ragloom generates deterministic point IDs from:

• canonical file path
• chunk index
• chunker strategy fingerprint

For the current filesystem source, the canonical document identity starts from the local canonical path and is then rendered into a stable file:// URI for sink payloads and doc_id derivation. For the reserved S3 source shape, the canonical document identity is defined directly as s3://{bucket}/{exact-key}.

This makes reruns safe.

The same file and same chunking config produce the same point IDs. Changing chunking parameters creates a new ID space, so old chunks are not silently overwritten by new content.

File-version identity is source-specific metadata hashed together with that canonical document identity. Today the filesystem source uses path, size, and mtime. The S3 design uses canonical S3 identity, object size, last-modified time, and normalized ETag so the planner and WAL can preserve deterministic replay semantics without requiring a per-object HEAD request.

Chunking

Ragloom supports several chunking modes:

By default, Ragloom runs in router mode and chooses a chunker by file extension:

• .md, .markdown, .mdx -> Markdown chunker
• .rs, .py, .js, .ts, .tsx, .go, .java, .c, .cpp, .rb, .sh -> code chunker
• other files -> recursive chunker

Useful flags:

• --chunker-mode router keeps extension-based routing
• --chunker-mode single --chunker-single recursive|markdown|semantic|code:<lang> forces one chunker
• --size-metric chars|tokens chooses chunk sizing mode
• --size-max, --size-min, and --size-overlap tune boundaries
• --enable-semantic enables semantic chunking in router mode
• --semantic-provider adapter|fastembed selects the semantic signal source
• --semantic-percentile <1..=99> tunes semantic split sensitivity when semantic chunking is active

Semantic chunking remains experimental and opt-in. It is only active with --enable-semantic in router mode or with --chunker-mode single --chunker-single semantic. When active, --semantic-provider and --semantic-percentile apply to both router and single semantic modes. In router mode, semantic chunking replaces the Markdown and generic text fallback paths while code extensions continue to use the code chunker. fastembed remains a feature-gated semantic provider and requires building with --features fastembed.

Indexed Payload

Each Qdrant point currently includes chunk text plus metadata such as:

{
  "canonical_path": "file:///Users/me/docs/intro.md",
  "doc_id": "doc_...",
  "tenant_id": "default",
  "file_extension": "md",
  "size_bytes": 842,
  "mtime_unix_secs": 1714300000,
  "chunk_index": 0,
  "total_chunks": 3,
  "previous_chunk_id": null,
  "next_chunk_id": "9c5eb1c7-f9a7-4fc2-a260-761842356849",
  "chunk_start_byte": 0,
  "chunk_end_byte": 842,
  "chunk_char_len": 842,
  "chunk_text_sha256": "sha256_...",
  "strategy_fingerprint": "markdown:v1|...",
  "chunk_text": "..."
}

The Qdrant point ID is the chunk identity and is not duplicated as a chunk_id payload field. previous_chunk_id and next_chunk_id contain those same deterministic point-ID values for neighboring chunks.

This is the part of Ragloom that makes inspection easier: you can look at a point in Qdrant and see where it came from, how it was chunked, and which neighboring chunks surround it.

Delete Synchronization

Ragloom tracks files it has previously observed under the configured source root. When a completed scan no longer sees one of those paths, the runtime plans a durable delete work item and the Qdrant sink removes all points with that document's stable doc_id.

Delete synchronization is idempotent and replay-safe: rerunning the same delete work is expected to leave Qdrant in the same state. Ragloom does not delete points for files it has never observed, and this is document-level cleanup only; it does not create, drop, or otherwise manage whole collections.

The same delete-sync model is reserved for S3: deletes are inferred only after a completed successful listing of the configured bucket and prefix. Missing objects from a partial or failed scan must not be treated as deletes.

For S3 object lifecycle semantics, Ragloom treats object identity as path-like:

• overwriting the same key is a new version of the same document when version metadata changes
• renaming a key is modeled as deleting the old key and ingesting a new document at the new key
• copying to a new key is modeled as a distinct document even if the content and ETag match

Observability

Ragloom emits tracing events for discovery, startup, embedding, Qdrant writes, and ingest completion summaries.

Environment variables:

• RAGLOOM_LOG_FORMAT=pretty|json
• RAGLOOM_LOG=info|debug|...

Example:

RAGLOOM_LOG_FORMAT=json RAGLOOM_LOG=info ragloom --config ./ragloom.yaml --openai-api-key "$OPENAI_API_KEY"

Ragloom does not log secrets, API keys, or full document contents.

Health endpoint

The local health endpoint is disabled by default. Enable it with either:

ragloom --config ./ragloom.yaml --health-addr 127.0.0.1:8080 --openai-api-key "$OPENAI_API_KEY"

or:

health:
  addr: "127.0.0.1:8080"

The address must be an IP socket address on a loopback interface, such as 127.0.0.1:8080 or [::1]:8080. Ragloom rejects non-loopback addresses to avoid accidentally exposing the operator endpoint outside the local machine.

Query it with:

curl http://127.0.0.1:8080/health

Ready responses return HTTP 200 and a small JSON body with daemon status and build/version information:

{
  "status": "ready",
  "ready": true,
  "version": "1.0.0-rc.1",
  "build": {
    "package": "ragloom",
    "version": "1.0.0-rc.1"
  }
}

Startup/bootstrap failures and fatal runtime-loop failures return HTTP 503 with ready: false and a short reason such as startup_failed or runtime_failed. The endpoint does not include document text, API keys, or full local paths.

Metrics endpoint

When the local health listener is enabled, Ragloom also exposes metrics on the same loopback address:

curl http://127.0.0.1:8080/metrics

The response uses the Prometheus text exposition format (text/plain; version=0.0.4) so local monitoring tools can scrape it directly. Metrics are numeric ingest and reliability counters only; they do not include document text, API keys, or full local paths.

Current metrics:

• ragloom_discovered_files_total
• ragloom_indexed_files_total
• ragloom_failed_files_total
• ragloom_emitted_points_total
• ragloom_pending_files
• ragloom_retry_attempts_total
• ragloom_retry_exhausted_total
• ragloom_retry_queue_depth
• ragloom_work_queue_depth
• ragloom_state_wal_bytes
• ragloom_state_failed_work_bytes
• ragloom_state_wal_pending_work
• ragloom_state_failed_work_pending

Durable-state metrics are numeric only:

• ragloom_state_wal_bytes reports the current WAL file size in bytes
• ragloom_state_failed_work_bytes reports the current failed.ndjson size in bytes
• ragloom_state_wal_pending_work reports the current count of durable WAL work items awaiting acknowledgement
• ragloom_state_failed_work_pending reports the current count of durable failed-work items still pending operator replay

These metrics do not expose document text, API keys, canonical paths, local paths, or failure-detail payloads. Ragloom updates them at startup and at durable state transition points; it does not re-parse the full journals on each metrics scrape.

For first-run validation, look for ragloom.ingest.summary. Ragloom emits it after an ingest window goes idle and again on shutdown when there is still unreported work. The summary stays structured and includes counters such as:

• discovered_files
• indexed_files
• failed_files
• emitted_points
• pending_files

Roadmap

v0.1 - First-run experience

Status: shipped in v0.1.1.

• example environment for local Qdrant setup
• clearer ingestion summary at runtime
• release binaries

v0.2 - More reliable daemon behavior

Status: shipped in v0.2.0, with restart-safe delete synchronization refined in v0.2.1.

• persistent local state
• bounded retry queue
• delete detection
• health endpoint
• metrics endpoint

v0.4 - Explicit platform expansion boundaries

Status: shipped in v0.4.1.

• keep the local-filesystem and polling-S3 ingest paths release-ready on supported Linux and Windows targets
• require green ci and quality-deep workflow states, or equivalent local maintainer verification, before cutting the release
• keep feature-gated paths such as fastembed exercised by dedicated checks without making them the default runtime path
• document PDF and DOCX extraction limits explicitly instead of implying broader parser guarantees
• keep remote-source scope narrow by supporting S3 only rather than expanding into a general remote-ingestion platform

v0.5 - State and recovery hardening

Status: shipped in v0.5.0.

• crash-safe explicit compaction for WAL and failed-work state
• direct compatibility with supported released v0.4.x state
• recovery contracts covering pending work, deletes, acknowledgements, and point identity
• state preflight checks and operator-facing replay summaries
• durable-state backlog and journal-size metrics

v1.0 - Stable operator contract

Status: release candidate in v1.0.0-rc.1.

• frozen point-ID, payload, CLI-default, and durable-state contracts
• direct upgrade coverage from released v0.4 and v0.5 state
• Rust 1.88 MSRV policy and supported-platform release gates
• live Qdrant collection, upsert, and delete smoke verification
• prerelease-safe GitHub Release and container publication

Limitations

Ragloom is intentionally small today.

Troubleshooting

Ragloom fails to start with Qdrant connection error

Make sure Qdrant is running and accessible:

curl http://localhost:6333/health

If using Docker, verify the container is running:

docker ps | grep qdrant

Collection not found error

If you started Ragloom with --create-collection-if-missing, it will bootstrap the target collection on first run.

For the default OpenAI model, this is enough:

cargo run --release -- \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --create-collection-if-missing \
  --openai-api-key "$OPENAI_API_KEY"

If you are using --embed-backend http or an OpenAI model Ragloom does not recognize yet, rerun with an explicit vector size:

cargo run --release -- \
  --dir ./docs \
  --qdrant-url http://localhost:6333 \
  --collection docs \
  --create-collection-if-missing \
  --collection-vector-size 1536 \
  --embed-backend http \
  --embed-url http://localhost:8080/embed \
  --embed-model default

If you prefer to manage Qdrant yourself, pre-create the collection before running:

curl -X PUT http://localhost:6333/collections/docs \
  -H "Content-Type: application/json" \
  -d '{"vectors":{"size":1536,"distance":"Cosine"}}'

Adjust the vector size to match your embedding model. Ragloom does not perform general collection lifecycle management beyond optional first-run bootstrap of the configured collection.

Empty or missing chunks

Check that your files are:

• UTF-8 encoded
• located somewhere under the configured directory
• regular files rather than symbolic links

If startup looks healthy but nothing appears in Qdrant, check the latest ragloom.ingest.summary event first. A non-zero failed_files or pending_files count usually narrows the problem down faster than scanning individual per-file log lines.

OpenAI API errors

Verify your API key is set without printing the value:

if [ -n "$OPENAI_API_KEY" ]; then echo "OPENAI_API_KEY is set"; else echo "OPENAI_API_KEY is not set"; fi

Test the embedding endpoint directly:

curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":"test","model":"text-embedding-3-small"}'

Do not share command output if it includes credentials or other sensitive response details.

Current limitations

• only local filesystem and polling S3 input
• only Qdrant as a built-in sink
• only UTF-8 text, Markdown, source code, text-extractable PDF loading, and deterministic DOCX text extraction
• no general collection lifecycle management beyond optional first-run bootstrap
• no broad persistent dead-letter queue or job-management subsystem

Contributing

Ragloom is maintainer-led and intentionally small.

Good contributions include:

• bug fixes
• tests
• documentation
• examples
• small focused connectors
• improvements to first-run experience

Please open an issue before starting large features.

Before opening a pull request, run:

cargo qa

Maintainers preparing release-sensitive or v1 support-boundary work should also run the authoritative deeper local gate:

cargo maintainer-qa

cargo qa stays the fast contributor gate. cargo maintainer-qa extends it with the checks that currently matter for local maintainer confidence:

• cargo test --workspace --features loom
• cargo build --features fastembed
• cargo test --workspace --all-targets --features fastembed
• cargo doc --workspace --no-deps with RUSTDOCFLAGS=-D warnings
• cargo deny check
• cargo audit

GitHub Actions runs the same deeper stability checks on pull requests.

See CONTRIBUTING.md for development expectations, SUPPORT.md for support policy, and SECURITY.md for vulnerability reporting.

License

Apache-2.0