All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- (config):
ConcurrencyConfig::max_concurrent_ocrand the--max-concurrent-ocrCLI flag set concurrent Tesseract recognition sessions on their own. Use it when the host has cores to spare but not the memory to run a recognition session on each of them. The value is applied as given: neither the thread budget nor the host's free memory reduces it, since both of those bound only the automatic limit. The first extraction in a process fixes the session count for the rest of that process, because the admission semaphore and the Tesseract handle pool that enforce it are built once and the pool's capacity is fixed when it is constructed; a later extraction that names a different value keeps the first one and logs a one-timeWARNnaming both numbers. Set the value on the first extraction, or run one process per value.ConcurrencyConfigis not#[non_exhaustive], so the added field breaks any Rust caller that builds the struct by literal without..Default::default(); such a caller must add the field or the rest pattern. Callers on every other binding are unaffected. (GH#1727)
- (pdf): scan detection and fabricated-mapping OCR routing now use the thread budget. Native PDF extraction graded every page for raster scan evidence, then read every page's text again to check its glyph-to-Unicode mapping provenance, and ran both passes one page at a time. The two passes held a single core for the whole document whatever
ConcurrencyConfig::max_threadswas set to, so no thread budget could shorten them. Both now run across the thread budget, sequentially on wasm32 which has no thread pool, and report the same per-page confidences and the same page lists in the same page order. Measured on a 32-core machine atmax_threads = 32, with the extracted output identical in every run: a 4778-page manual from 66.9 s to 63.3 s, and a 676-page book from 24.1 s to 21.9 s. The two passes are a small share of a native extraction, so the wall-clock gain is bounded by that share. (GH#1723) - (ocr): concurrent Tesseract recognition follows the thread budget instead of a fixed four. The limit was a compile-time constant that no configuration reached, so raising
max_threadscould not raise recognition throughput, and recognition is most of the run on a scanned document. The default is now the thread budget, reduced to the number of sessions the host's free memory holds. Setmax_concurrent_ocrto4to keep the previous behaviour. (GH#1727) - (pdf/ocr): the per-page OCR route sizes its batch by what a page costs to hold, not by the limit on returned content size.
max_content_sizebounds the text an extraction returns and says nothing about the rasters a batch holds while it works, so on an ordinary 150 DPI document the arithmetic pinned the batch at four pages and no stage of the route could use more than four threads whatever the thread budget said. The width now comes from the page box and the render resolution, measured against free memory less the document and a reserve. Each page's own peak is still checked againstmax_content_size, one page at a time. Peak resident memory rises with the wider batch -- 2,234 MB to 3,583 MB on a 731-page document at a 32-thread budget -- because more rasters are in flight. (GH#1724) - (pdf): the embedded-image pass now runs across the thread budget. Configuring OCR switches whole-document embedded-image extraction on, and that pass walked the document one page at a time, so no thread budget shortened it. Pages now decode in parallel (sequentially on wasm32, which has no thread pool), in the same order, and the returned bytes are unchanged. Measured on a 731-page document holding 2,881 images at a budget of 32: the pass falls from 20.1 s to 2.2 s, and the whole extraction from 57.3 s to 41.3 s. Peak resident memory rises with the budget, because each pool thread holds one page's raw pixel buffers and their PNG re-encodes at once, where a single page's were live before: over the pass itself, 1,133 MB before against 1,463 MB at a budget of 8 and 2,186 MB at 32. The pass carries no memory budget of its own, so the ceiling is the thread budget, which is
min(cpu_cores, 8)untilConcurrencyConfig::max_threadsraises it. The pass also no longer decodes and PNG re-encodes a full-page image on a page that already carries native text when OCR is the only reason the pass is running at all:should_skip_pdf_image_ocrexcludes exactly that image from OCR anddrop_ocr_only_imagesthen drops it from the result unread, so the decode was pure waste. That image still appears in the result with its dimensions, bounding box, page number and alt text intact and an emptydata; a document where any other consumer -- image extraction, captioning, QR codes, inline-image OCR, page rasters, or styled HTML rendering -- would actually read the bytes is unaffected. (GH#1732) - (ocr):
candle-deepseek-ocrreads device tensors back in one transfer instead of one element at a time. The mixture-of-experts gate copied its score matrix per expert, per row, per layer, per generated token, and the SAM relative-position lookup read its index inside a nested loop per attention layer, per crop. Both are now a single bulk read. Measured on an A100 at BF16: 62.71 s to 8.09 s per page (7.8x), output byte-identical, peak memory unchanged. A 22-page scanned document that previously exceeded the 600 s extraction ceiling now completes in 322 s. (GH#1711, GH#1714) - (ocr):
candle-deepseek-ocrresizes the relative-position table and scatters the image embeddings as device operations instead of per-element loops. The resize sampled its axis one position at a time and concatenated one tensor per output position, on each of the four global-attention layers of the batched 640 px local-crop pass; it is now candle's bilinear resize over a one-row image, which is the half-pixel sampling the reference implementation uses. The scatter gathered rows one at a time and stacked one tensor per sequence position, once per page; it is now two gathers and a select. Measured on an NVIDIA L4 at BF16 over a full Letter page, five repetitions: 47.610 s to 47.546 s per page, which sits inside the spread of the unchanged arm, so this is not a speedup on that workload. Both arms decode the same 2,052 characters. Peak GPU memory falls from 8,818 MiB to 8,722 MiB. (GH#1719) - (ocr): the per-page OCR-image render/encode size check now runs through one shared helper instead of three separate copies. GH#1724 and GH#1731 each fixed a route that summed a whole page batch against
security_limits.max_content_size-- a limit its own error text names as a per-image bound -- rejecting every page in the batch once the sum crossed it. Both fixes, and the mixed native/OCR route's own single-backend path, now call onevalidate_png_encode_pages_individuallyhelper that charges each page on its own, so a new call site cannot reintroduce the batch-summing shape by calling the lower-level batch-peak function directly. No behaviour change on any of the three existing routes. (GH#1748) - (pdf): a
/Fontdictionary shared across pages no longer redoes TrueType cmap donation on every page that touches it.share_truetype_cmapsclones and mutates each undonated font withArc::make_mut, and the font-set caches deliberately store each dictionary's pre-donation fonts (#1725 made that order-independent), so every cache hit re-ran that clone for the same font. Donation among fonts within one dictionary's own resolved set is now cached alongside the pre-donation set, under the same key, so it depends only on that dictionary's own resources and not on which page or Form XObject was read before it. Donation whose donor lives in a different dictionary (a Form XObject donating to its page, say) is unaffected and still runs on everyload_fontscall. (GH#1746) - (pdf): fabricated-mapping OCR routing no longer reads every page's text a second time. The main text pass and the provenance pass each read every page of a native PDF: the main pass to assemble content, the provenance pass (GH#1723) to grade each page's glyph-to-Unicode mapping. Neither carried its read into the other, so a 731-page document spent 14.2 s of a 24.6 s extraction on the two passes together. The main pass now keeps each page's fabricated-character counts from the raw spans it already reads, and the provenance pass consumes those counts instead of reading the page again; a document with default-off optional-content layers still reads separately, since the main pass reads layer-filtered spans there and the two would otherwise disagree. Retaining the counts holds page text already held for content assembly, not rasters, so peak memory is unaffected.
fabricated_text_pages, scan confidences and extracted content are unchanged. (GH#1744) - (pdf): scan detection no longer decodes an embedded image's pixels just to classify its codec. Grading a page for raster scan evidence decoded every embedded image on the page in full -- the same decode the embedded-image extraction pass repeats moments later when OCR is configured -- only to read whether the result was JPEG or CCITT and how large its bounding box was. Both are already known from the cheap Phase 1 handle enumeration that walks the content stream without decompressing anything, so scan detection now reads bounding box and filter chain from there instead. The embedded-image extraction pass, the only remaining caller that needs decoded pixels, is unaffected; classification output (image area ratio, codec class) is unchanged for every image that decodes; an image the decoder rejects now counts toward the page's image coverage where it was silently dropped before, and images under the 8 x 8 px extraction floor stay excluded. (GH#1732)
- (pdf): layout detection no longer fails partway through a long document. The layout pass charged every page raster it had already produced against
security_limits.max_content_size, so the running total crossed the 100 MiB default after about twelve standard pages, whatever the page size. Layout detection then failed for the whole document while extraction carried on, and the caller got a successful result with no layout hints and a warning naming the page's pixel dimensions, which reads as a page-size limit and is not one. Each batch is now charged on its own, so the limit bounds the work in flight rather than the length of the document. The layout pass still retains every chunk's raster, andsecurity_limits.max_content_sizeno longer bounds that retained set; onlysecurity_limits.max_pagesdoes, and it is unset by default. (GH#1721) - (ocr): the
autodevice preference reports when it cannot reach the accelerator, instead of running the whole job on the CPU in silence. A failed CUDA or Metal init discarded its error and returned the CPU device with no log line naming the cause, so the run looked like a hang: the GPU stayed at 0%, several cores were busy in the forward pass, and the last log line was whatever ran before it. A warning now names the accelerator and the underlying error. GLM-OCR, PaddleOCR-VL and TrOCR already resolved the device once per engine, inside the engine pool's cold start;candle-deepseek-ocrnow does the same, instead of resolving -- and warning -- once per page. A probe that fails is retried on the next page rather than remembered, so a transient accelerator failure does not pin the process to the CPU for the rest of its life. (GH#1722) - (pdf): a row-padded image buffer is rejected instead of panicking.
ImageBuffer::from_rawrejects a buffer that is too small and accepts one that is too large, keeping the extra bytes. A decoder that pads each scanline to an alignment boundary produces exactly that, and the mismatch surfaced inside the PNG encoder as its own size assertion, so extraction panicked instead of returning an error the caller could act on. Measured on a 229x265 RGB image whose buffer held 182,320 bytes against the 182,055 the image needs: one byte of padding per row. The exact length is now checked before construction for all three pixel formats, returning the same recoverable error an undersized buffer already returned.to_png_bytesinxberg-native-pdfalready carried this guard; it was never applied to this call site. (GH#1735) - (pdf): a page's extracted text no longer depends on the order pages are read. Reading a document's pages concurrently could return different text on different runs. Sequentially, a page whose
/Fontdictionary is written inline under the same resource names as an earlier page's decoded through that earlier page's fonts, so where the two fonts differed the later page read wrong. Extraction now returns the same text every time, whatever order the pages are read in, and each page decodes through the fonts its own resources name. (GH#1725) - (ocr): OCR with layout detection no longer drops every page of a batch. The route that OCRs pages a layout pass has already rendered charged the whole batch's render-and-encode peak against
security_limits.max_content_size, a ceiling that bounds one image. Batch width on that route is the resolved thread budget, so from five US Letter pages up at the default 150 dpi the sum crossed the 100 MB default and the OCR run was refused outright; the automatic OCR fallback then returned the document's native text with a warning, which reads as a successful extraction that is missing its OCR'd text. Each page is now charged on its own, as the limit's name says. The batch's own footprint is bounded by the thread budget alone and not bymax_content_size, so peak live bytes on this route rise with a wider budget. (GH#1731) - (pipeline): embedded images populated without OCR are no longer dropped from
images. The GH#1703 fix (1.2.6) dropped every embedded image's bytes after extraction unless the caller had asked for image extraction, captioning, QR codes, inline-image OCR or page rasters. It ran on every extraction, including ones with no OCR configured at all, so a Markdown inline SVG data URI, a Jupyter output or attachment image, or an ODT/DOCX/PPTX embedded picture came back withimagesempty even though nothing had read those bytes for OCR. The drop now runs only when OCR was configured to run on embedded images, which is the case GH#1703 addressed. (GH#1703) - (ocr):
candle-glm-ocrresolves its layout model once per process, not once per page. Each page re-ran the Hugging Face path resolution and checksum verification of the 131 MB PP-DocLayout-V3 model; that is now cached by directory, the same shape the layout engine's sibling models use. Measured on 22 pages: 82.71 s to 6.24 s. A failed resolution is not cached, so the next caller retries. (GH#1718) - (pdf): a document whose text-plausibility check could not judge any page now says so.
implausible_text_pages: []meant either that every page was checked and passed or that no page held enough prose to be checked at all (contracts, invoices, forms, agenda packets), and a caller could not tell the two apart. A document where no page could be judged now carries a processing warning naming how many pages were examined; the warning is suppressed when OCR was already forced. Extraction behaviour and OCR routing are unchanged. (GH#1709) - (pdf): a numbered heading that wraps twice is no longer closed after its second line. The paragraph grouper's heading-wrap exemption only recognised a heading that had absorbed exactly one wrap (
visual_line_count(¤t_lines) == 2), so a heading spanning three or more visual lines was cut before its own last line: the orphaned line was then welded to the body paragraph beneath it, with no signal left to separate the two on a two-column page. The exemption no longer counts wraps; it now applies at every line while the paragraph is still nothing but the heading and its accepted continuations, so a heading is closed at the first line that genuinely fails to continue it, however many wraps came before. (GH#1740) - (pdf): a numbered heading set as one
TJarray with a kern for the tab no longer absorbs the other column's line at the top of a two-column page. When a producer places a heading's marker and title in oneTJarray with a kern standing in for the tab ([(3.)-1329.5(Title )] TJ), the kern becomes a space-only span that is always regular weight regardless of the surrounding bold context, which broke the reading-order heading-run detector's clustering right at the marker/title boundary. The narrower run this produced no longer covered the marker's column position, letting an unrelated span from the other column land inside the heading. The same page set with the marker in its own text object (no kern) already read correctly; it now reads the same way regardless of how the producer set the marker. (GH#1738) - (config): the configured thread budget is no longer silently dropped when two extractions start together.
init_thread_poolsbuilt the process-wide Rayon pool outside thecall_oncefence that guards it, so a second concurrent caller could be released -- with the atomics already set -- before the pool existed. If that caller reached its own parallel work before the first caller'sbuild_global()finished, it silently installed Rayon's default pool first, and the configuredmax_threadswas never applied for the life of the process. The pool now builds inside the same fence, so no caller observes the installed limits before the pool they describe actually exists. (GH#1750) - (pdf): the dense two-column repair no longer lets a table's own grid vote for the page's gutter. On a two-column page that also carries a table,
detect_split_xcounted a table row's internal cell gaps as gutter evidence -- a 5-column table's rows outvoted the page's real two-column lines and placed the split inside a column, and the same votes fed the hanging-label snap, so a numeric table column straddling the true gutter could be mistaken for a stack of hanging clause numbers and pull an already-correct split back into the table. Both now skip a line whose inked spans open four or more internal gaps, the shape of a table row of five or more columns; a two-column line with a hanging number on each margin opens three, so three would exclude the very lines that carry the gutter. Separately,both_sides_are_columns(gating the split's widened-corridor rescue) required both sides of a candidate gutter to classify as prose, stricter than the per-band reorder gate it feeds, which already accepts one non-prose (table) side when the two sides do not pair up row for row; it now applies the same test. A third gap remained even with both of those in place:corridor_is_hanging_label_indentread a table's own narrow edge column (cells stacked hard against the gutter's left wall) as a hanging clause number, so the widened corridor rescue still refused the page's real gutter and left the split inside the table. It now also requires an inked span on the far side of the corridor, on the same line, before counting a candidate -- a hanging label always has its clause text there; a table's edge cell never does, since the row's other cells sit on the label side and whatever text starts past the corridor belongs to an unrelated, unpaired line. Fixes the reporter's own page 1 (a five-column table with a narrow last column, plus a centred page number in the gutter); a table filling a whole column with no row-pairing across the gutter (the reporter's page 4) is fixed too. There the wrong split is crossed by only one line -- an introductory paragraph above the table, not the table itself -- because every table row's own evidence that the split sits inside it is its internal cell gap, never a span crossing the split, so the count that decides whether to widen the corridor search never reached its threshold and the search that would have found the true gutter never ran. That count now also includes a line whose split evidence is its own internal, multi-column cell gap rather than a literal crossing, so the search runs and the already-fixed corridor guards find the true gutter the same way they do on page 1. (GH#1742)
- (pdf/ocr):
OcrQualityThresholds::enable_plausibility_ocr_routingandOcrQualityThresholds::min_reliable_language_chunk_ratiocontrol language/dictionary-plausibility OCR routing, andPdfMetadata.implausible_text_pagesreports which pages a page's decoded text failed to read as any real language. (GH#1696) - (config):
ExtractionConfig::runs_ocr_on_embedded_imagesandExtractionConfig::wants_own_bytes_in_resultare exposed on every binding, beside the existingneeds_image_data. The first is the predicate the pipeline uses to decide whether a container's embedded images are OCR'd; the second is the pre-GH#1662 formula (extract_images, captioning or QR codes) that decides whether a standalone image's own bytes are echoed intoimages. (GH#1662)
- (ocr): the
ocr,ocr-wasm, andocr-pipelineCargo features now implylanguage-detection. The language/dictionary-plausibility OCR-routing signal (GH#1696) is gated the same as the rest of the OCR module and would otherwise be a silent hole under a build that enables onlyocr-pipeline(the VLM-only, Tesseract-free pipeline). whatlang (behindlanguage-detection) depends only onhashbrown, so this is wasm/Android-safe. (GH#1696)
- (ocr): OCR no longer returns every embedded image's raw bytes. Image bytes read only to feed embedded-image OCR (GH#1662) are dropped once OCR has consumed them and any OCR text it produced has been rendered into
content;imagesnow carries data only when the caller asked for image extraction, captioning, QR-code detection,pdf_options.ocr_inline_images, orimages.include_page_rasters;counts.imagesstill reports how many embedded images the document had. (GH#1703) - (pdf): a
/ToUnicodemapping that resolves to the wrong-but-real letters is now routed to OCR too. GH#1667 fixed the case where a page has no usable mapping tier at all (MappingProvenance::Fallback); a page whose/ToUnicodeCMap DOES resolve, but consistently to the wrong letter (e.g. a ROT-shifted mapping), passed every character-shape check unchanged and OCR never fired, because the text is structurally indistinguishable from real prose. A language/dictionary-plausibility signal now flags such a page under the defaultAutostrategy, and the opt-inScannedPagesstrategy picks it up through the existingscanned_pagesunion. A page with no explicitocrconfig keeps its native text, but a warning now names the defect. Closes the remaining gap in GH#1667. (GH#1696) - (pdf): the mixed native-and-OCR path no longer drops every native table on a page OCR never touched. On a long document where only some pages needed OCR, the mixed path replaced the whole document's table list with just the OCR pages' tables whenever OCR found even one, silently dropping every native table on every other page. Only the tables of pages OCR itself produced a table for are now replaced; a page the mixed path never sent to OCR keeps its native tables. (GH#1670)
- (pdf):
include_document_structurealone now triggers the structured native extraction pass. Previously the structured pass ran only for a Markdown/Djot/HTML/DocTags output format, an explicit hierarchy config, inline-image OCR, or a content filter -- never forinclude_document_structureitself. A caller who set only that flag, with output left at itsPlaindefault, silently got a flat, paragraph-only document, so the structure tree it asked for came back holding nothing butparagraphnodes even though the extractor found headings, tables and page breaks. This configuration now renders its plain-text output from the structured document, so the renderedcontentchanges with it: table rows render one per line, and headings and page footers reflow. A caller that sets only this flag and depends on the previousPlaintext sees that text change. The table count is unaffected. (GH#1668) - (docx): a paragraph inherits the list numbering that its style carries. A list whose paragraphs hold no direct numbering and take it from the paragraph style instead, which is the shape Word writes for its built-in list styles, came out as plain paragraphs with no list markers. The parser now resolves numbering through the style's
basedOnchain. Direct paragraph numbering still takes precedence over the style. (GH#1663) - (pdf): the fabricated-glyph-mapping provenance lookup is repaired, and a page it flags is sent to OCR. The extractor read each text run's mapping provenance after it had renamed the run's font to the resolved
/BaseFont, but the provenance table is keyed by the raw resource alias, so the lookup missed for every font whose base name differs from its alias and the signal was empty almost everywhere. The lookup now runs in the same pass and uses the alias, and a page whose text carries no usable mapping tier at all (MappingProvenance::Fallback) now forces the OCR fallback under the defaultAutostrategy. A font whose mapping resolves, even to the wrong characters, is a different failure mode this signal does not cover. (GH#1667) - (ocr): a low OCR confidence now lowers the reported quality score. The quality score measured only how clean the text looked, so a page that OCR itself had little confidence in still scored as clean whenever the characters happened to be well formed. The score is now capped by the word-count-weighted mean of per-page OCR confidence, once at least 20 words were recognized -- the same fold
ExtractionConfidence::ocr_aggregateuses, shared rather than recomputed, though the aggregate itself reports on any recognized word and so can differ below the 20-word floor. A native, non-OCR extraction is unaffected. (GH#1669, GH#1694) - (ocr): a cold start no longer builds one candle OCR engine per waiting caller. Each candle backend's engine pool looked for a cached engine, released the lock, built the engine, then stored it, so every request that arrived during the first build loaded its own copy of the model and a burst could exhaust the GPU memory. All four backends and the GLM-OCR layout pool now use the crate's engine cache, one per backend, which holds the write lock across the build: the first caller builds and the rest wait for that engine. Loads of different engines on the same backend (two TrOCR variants, say) therefore also serialise during a cold start. A failed load stores nothing, so the next caller retries. (GH#1683)
- (ocr): the aggregate OCR confidence is populated for page-level results, and both routes weight it the same way. A document whose OCR confidence arrived one score per page left the aggregate empty, so a caller that read it got nothing back. Both routes now fold the scores by recognized word count instead of by element, so a page of many short lines no longer counts for as much as a page of few long ones. This changes the number the embedded-image route already reported. (GH#1677)
- (ocr): the Tesseract language probe run at backend registration now finds the installed languages. The probe initialised Tesseract with an empty datapath, which makes the library look one
tessdatadirectory deeper than the resolver real OCR jobs use, so on a layout where jobs foundeng.traineddatadirectly the probe logged "couldn't load any languages" and silently fell back to a hardcoded language list. It now resolves the tessdata directory the same way a job does. (GH#1671) - (ocr): a page whose embedded-image OCR retry came back empty says so. When page-raster OCR failed and the embedded-image retry ran but returned nothing -- a candle VLM backend reports success with empty content -- the page warning named only the first failure, with no trace that a retry ran at all. The per-page warning and the wholesale "every page failed" error now report the retry outcome alongside the first failure. (GH#1673)
- (ocr): OCR throughput scales with the thread budget. The mixed native-and-OCR route rasterised each batch's pages one at a time regardless of the configured thread budget, so a 240-page document with 80 scanned pages took the same 9 s at 4, 8, 16 and 32 threads. Pages now render in parallel across the batch (sequentially on wasm32, which has no thread pool), in the same order as before; measured 8.8 s to 3.3 s at 4 threads and 9.0 s to 1.0 s at 32. (GH#1666)
- (ocr): a DOCX, PPT, PPTX or HTML embedded image is read before OCR runs on it. The predicate that decides whether a container reads an image out of its archive did not count embedded-image OCR, so with OCR configured and
run_ocr_on_imagesleft on, the image was attached with an empty buffer and OCR reported "Could not determine image format" for zero bytes. A standalone image extraction'simagesoutput is unchanged. (GH#1662) - (ocr): the mixed native-and-OCR route sizes its render batch against
security_limits.max_content_size. The batch was sized from the resolved thread budget alone, so a wide thread budget requested a batch whose estimated PNG-encode peak crossed the fixed content limit, and every page in the rejected batch was skipped rather than the extraction failing. The batch is now capped up front from one representative page's estimated encode cost at the effective render DPI; the real peak is still checked, and still rejected if genuinely too large, afterwards. (GH#1665) - (ocr): the candle OCR backends no longer reject every rendered PDF page. The PDF OCR route stamps
source_dpiandpage_rotation_degreesinto the sharedbackend_optionsfor every backend, and the candle backends (candle-paddleocr-vl,candle-glm-ocr,candle-deepseek-ocr) deserialise their options strictly, so every rendered page failed validation and came back empty. Exactly those two pipeline hint keys are stripped before the candle options are read; any other unknown key still fails validation. (GH#1672) - (ruby): a tagged enum's payload readers return the stored value. The generated
Datareaders calledsuperon a class that has no such method, so reading a payload field on a tagged-enum value raisedNoMethodError. Regenerated on alef 0.93.1, whose Magnus emitter reads the stored member. (alef 0.93.0) - (ocr): a candle VLM backend no longer emits implausible lines of noise.
candle-paddleocr-vland the other three candle backends would occasionally emit a CJK-script run in an otherwise English document, or bare LaTeX markup on a plain-text OCR task, most visibly on a non-text region such as a handwritten signature scribble. A line whose classified letters are mostly in a script the configured OCR language list does not cover, or whose content is mostly\command-shaped LaTeX tokens with no bare word on a plain-text task, is now dropped, with a warning naming how many lines were removed; a$$/``` fenced block and a genuine formula/table/chart task are exempt. (GH#1676) - (ocr):
candle-deepseek-ocrreads the whole page. Generation was capped at 128 new tokens (~400-650 characters, the 15-25% observed), the page was fed as a single 1024 px global view, and weights were loaded as F32 (16.5 GB on a 24 GB card). The cap is now a model configuration field defaulting to 4096 in line with the other full-page backend, pages larger than 640 px are also tiled into 640 px local crops as in the reference "Gundam" mode, the dtype follows the device (BF16 on CUDA, F16 on Metal, F32 on CPU), the weights auto-download from a checksum-pinneddeepseek-ai/DeepSeek-OCRrevision somodel_pathis optional, and a run that starts looping (the repeated-digit table on a signature block) is detected once the repeat fills a trailing detection window and truncated back to a single copy of the repeated unit, rather than burning the rest of the token budget on it. (GH#1674) - (ocr):
candle-glm-ocrno longer garbles hex identifiers such as UUIDs. The repetition penalty defaulted to 1.1 with no upstream basis (zai-org/GLM-OCR'sgeneration_config.jsonsets none) and was applied once per occurrence of a token in the whole decode history, so a token repeated k times was suppressed by1.1^k-- exponential enough that hex digits and-inside a long identifier lost probability mass until argmax drifted onto a re-emitted group. The default is now 1.0 (a no-op, matching upstream), and each token in the trailing 64-token window is now penalised exactly once regardless of how many times it recurs. (GH#1675) - (ocr):
candle-deepseek-ocrno longer runs its prefill without a causal mask.prepare_causal_attention_maskbuilt a 0/1 keep-mask that the attention step added straight onto the logits, so a future token was nudged down by one instead of blocked and the prompt-and-image prefill (and the KV cache it seeds for every later token) saw the whole sequence; the mask is now the additive 0/-inf form GLM-OCR and PaddleOCR-VL use. This is the one DeepSeek-only forward-pass defect the GH#1701 audit found; with it fixed a full Letter page decodes without repetition on Metal at both F16 and F32, but the reported CUDA F32 loop has not been re-run. (GH#1701) - (ocr):
candle-deepseek-ocrno longer fails on the first 640 px local crop. Resizing the SAM relative-position table to the crop size multiplied each row by a one-element tensor, which candle does not broadcast, so every page large enough to be tiled failed withshape mismatch in mulbefore a token was decoded; the resize now scales withaffineand samples at half-pixel positions like the referenceF.interpolate(mode="linear"). Verified on Metal at F16: a 2,018-character Letter page comes back whole. (GH#1674)
1.2.4 was tagged but never reached a package registry: its generated binding files still named 1.2.3, so its publish run was cancelled. 1.2.5 is the first published release carrying the 1.2.4 changes below in addition to its own.
- (python): a config field holding a nested data-carrying enum is no longer dropped by
from_json/to_json. The Python DTOs marked every field whose type transitively held a data enum as#[serde(skip)]--CaptioningConfig.llmandChunkingConfig's nested options among them -- so a JSON payload round-tripped through such a config silently lost those fields. Regenerated on alef 0.92.1, whose Python emitter no longer treats a serializable enum wrapper as opaque. (alef#394) - (ppt): a legacy
.ppt's embedded OLE objects are extracted. A Word document or Excel sheet inserted as an object -- the way PowerPoint 97-2003 decks routinely carry a table -- lives in the deck'sExOleObjStgrecords, which the extractor walked over as opaque bytes, so the slide came out as its title and nothing else. The.pptxpath already recursed intoppt/embeddings/; the legacy path now does the same: each embedded object's storage is resolved through the persist chain (so a superseded save's copy is never read), decompressed when the record says so, identified by its root stream (WordDocument,Workbook/Book,PowerPoint Document, or an OPCPackage) and extracted intochildrenasslide<N>/oleObject<id>.binwith the slide whose shape displays it. The decompressed size is bounded bymax_embedded_file_bytes(falling back tomax_archive_size), the object count bymax_files_in_archive, andmax_archive_depth = 0disables the recursion, as for.pptx. Metafile presentation pictures (EMF/WMF) are still not emitted as images. (GH#1660) - (pdf): a ruled table drawn one bar per cell no longer loses all but its last two rows.
Word draws cell borders as per-cell filled bars with a corner square at every crossing, so
every vertical rule terminates at every horizontal rule. The section-divider split, which
reads a full-width rule that no vertical rule runs through as the boundary between two
stacked form sections, therefore treated every internal rule of such a table as a divider,
cut the table into one-row pieces and discarded the pieces below the cell minimum -- a
12-row table came back as
11 | 290,12 | 300. A rule that every vertical rule stops at carries no section information when the same is true of every other rule on the table, so a table whose internal rules all qualify is now left whole; a form whose sections' rules stop at a minority of the rules still splits. (GH#1656)
- (llm):
LlmConfigexposes liter-llm's provider response-byte cap. liter-llm'sClientConfigBuilder::max_response_bytesbounds every HTTP response body read from a provider, but xberg'sLlmConfignever forwarded it, so downstream products had no way to bound provider responses.LlmConfig::max_response_bytesmirrors it: it bounds response bodies on every non-streaming call and the error body read on a failed request (a successful streaming response keeps its own existing frame bounds), and defaults toNone(unbounded), matching liter-llm.Some(0)is rejected byLlmConfig::validaterather than reaching liter-llm's own builder. (xberg-io/xberg-enterprise#1568, xberg-io/xberg-enterprise#1861)
- (pdf): a page-sized background rectangle no longer seeds a whole-page table. Many
Office-to-PDF producers draw a white rectangle over the entire page before anything else.
That rectangle passed the table-primitive filter (its
< 1000 ptbound does not exclude an A4 or Letter page) and, because clustering unions any two primitives whose boxes intersect, pulled every rule on the page -- the real table's borders, the footer rule, a figure's frame, strokes inside a drawing -- into one cluster with the page as its box. The cluster fallback then built a table over the whole page: the heading above the real table became its first row, torn at the table's own column rule, and the first words of the prose below it became its last row, with those words missing from or doubled in the paragraphs that followed. A rectangle covering at least 90 % of the page's MediaBox in both dimensions is now treated as page furniture and dropped before clustering, so the real table's rules cluster on their own and the page comes out as it does without the background. A full-page-width rule (one dimension at page scale) is deliberately still a primitive. (GH#1656) - (pdf): a single-column page of hanging-number headings is no longer read as two columns.
Some producers (Distiller among them) emit the tab between a heading's number and its title
as a space-only span in a different font, and leave space-only spans for blank lines. The
dense-two-column repair counted those spans as column population and let a blank line's two
whitespace spans and a footer split across both margins vote for a gutter, so five headings'
tabs plus two lines of nothing met the six-line quorum and the page was emitted
column-major:
6,6.1,6.1.1as bare numbers and their titles as separate unnumbered headings. Whitespace-only spans no longer count toward the per-side density gate, the gutter vote, the whitespace corridors or the row-pairing guard, and a per-line gap wider than a quarter of the page width is no longer accepted as gutter evidence (a blank line's or a footer's gap; every real gutter measured here is under 10 %). (GH#1655) - (ocr): a configured
security_limitsnow reaches the Tesseract image decode. GH#1554 routedExtractionConfig::security_limitsontoOcrConfigbefore each OCR call, but the Tesseract backend'sconfig_to_tesseracthad no field to carry it into and built a fresh defaultExtractionConfigfor the processor, so every Tesseract decode -- standalone images, embedded images, scanned pages and the targeted page fallback alike -- ran under the 100 MiB default no matter what the caller set. A 600 DPI A4 scan was refused with an error naming104857600 byteswhile both configuration objects said 5 GiB. The internalTesseractConfignow carries the limits (outside the cache key: they gate whether a decode runs, never what it produces), the processor prefers them over its synthetic config, and the standalone-image and PDF embedded-image routes inject the caller's limits like the other routes already did. A limit set directly onOcrConfigis honoured whenExtractionConfigcarries none;ExtractionConfigstill wins when both are set. (GH#1651) - (go):
Register*no longer leaks itscgo.Handlewhen the C vtable allocation fails. EveryRegister*wrapper in the Go binding created the handle before allocating the C vtable, and the allocation-failure branch returned without deleting it, keeping the bridge and the caller's implementation reachable for the life of the process. The branch is only reachable when a smallmallocfails, so this is a resource leak on an error path rather than an exploitable defect; it is fixed in the generator (alef 0.91.6) and regenerated here. Reported by @OvOhao (GHSA-q5pq-8g86-v9j9). - a table's multi-word cell no longer bleeds a trailing word into the next column, a
right-aligned amount column split by digit-width drift is folded back together, and a
legitimately sparse but independently-headed column (or a sparse first data row) no longer gets
the whole table rejected. Reconstructing a table from word boxes decided each word's column
independently by nearest x-position, so a wide cell's own trailing word (e.g. a long
description's last word) could resolve to a neighboring column's anchor instead of its own
cell's. Column membership for a data row is now decided once per merged cell cluster and applied
to every word the cluster contains; a drift-split right-aligned numeric column (mutually
exclusive per row, no independent header of its own) is folded into one column; and a section
caption sharing a table region with its real header row is dropped. This cell-clustering and
column-assignment logic lives in the shared
table_coremodule, so it also affects native-PDF word-position table reconstruction, not just OCR. Separately,pdf's shared table post-processor no longer treats an independently-headed but infrequently-populated column (e.g. a bank statement'sDEPOSIT, populated on a minority of transaction rows) as noise, and no longer folds a sparse first data row (one missing an optional numeric field) into a bogus multi-row header merge with the row after it -- and that header-shortcut no longer stops one row early on a genuine two-row text header, which also fully populates a digit-free first row. The punctuation cell-merge gap this widens for a lone dash/colon glyph is now conditioned on the punctuation genuinely bridging two neighboring words on both sides, so an isolated punctuation cell near a real column boundary (a-placeholder, a:) no longer fuses two columns' content into one cell. (GH#1649)
- (ocr): the registry reports each backend's declared languages.
list_ocr_backends()returned only names, so a caller validating an OCR language against the installed backends had to maintain a second language table of its own.list_ocr_backend_capabilities()returns oneOcrBackendCapabilitiesrecord per registered backend — its name and the languages it reports — ordered by backend name. An emptysupported_languagesmeans the backend does not enumerate its languages, not that it supports none, soocr_backend_supports_language()is provided for callers that need a decision rather than a list. (GH#1643) - (ocr): mixed-page PDF extraction reports one coordinate frame per OCR'd page. Public
ocr_elementscarry the OCR backend's raster coordinates, but the per-page processed raster size lived only in the page-local result and was discarded before the elements reached the final document. A multi-page consumer was left with the document-levelocr_processed_image_width/ocr_processed_image_height, which is not authoritative for pages of differing size, preprocessing, or rotation, and so could not normalize a box safely.metadata.additional.ocr_page_coordinate_framesnow carries one record per OCR page that has public elements —page_number,width,height,unit: "pixel",origin: "top_left"— ordered by page number and joined to an element through the element's ownpage_number. No record is emitted for a page whose raster dimensions are absent or invalid. (GH#1645) - (ner): Rust callers can share xberg's process-wide GLiNER backend cache.
text::ner::gline::get_or_init_backendperforms model initialization on Tokio's blocking pool, returns the sameArcfor the same model and thread budget, and leaves failed initializations retryable. - (pdf): every hierarchy page now advertises the coordinate frame its boxes live in. Hierarchy
bounding boxes are emitted in raw PDF user space, but nothing named that space, so a consumer
could not tell whether a box needed translating by a non-zero MediaBox origin or rotating for a
page's
/Rotate.metadata.additional.pdf_page_coordinate_framesnow carries one record per page that has hierarchy blocks —page_number,origin_x/origin_y(the MediaBoxllx/lly, which may be non-zero and negative),width/height,unit: "point",origin: "bottom_left", andclockwise_rotation— ordered by page number.width/heightare the MediaBox extent and are deliberately not swapped for a 90/270 rotation, because they describe raw user space rather than the displayed frame; this is the opposite convention toocr_page_coordinate_frames, which reports an already-rotated raster, so the two must not be assumed to agree for the same page. A page is omitted entirely, rather than reported with a default, when its/Rotateis present but not a valid multiple of 90 or its MediaBox yields no usable extent. (GH#1653, GH#1654)
- (ocr): an isolated blank quantity cell in an invoice table can be recovered from its own
pixels. A table whose
QTYcolumn has strong integer support — at least two recognized whole-number quantities — but exactly one blank cell now gets a single targeted retry that re-reads only that cell's bounded pixel region in single-line segmentation mode. The retry never derives a quantity from a price or total column, and any table outside this narrow shape takes the existing fast path unchanged. A header split across two words (UNIT/PRICE,LINE/TOTAL) is also reassembled into one header cell when the fragment sits nearer a data-supported column than any standalone column of its own. This OCR quality change ships without a corpus-wide A/B measurement. - (reranker): a cancelled
rerank_asyncno longer releases its concurrency permit while its inference is still running. The permit was held by the awaiting future rather than by thespawn_blockingtask, and a blocking task cannot be cancelled — dropping its handle merely detaches it. A caller that timed out or dropped therefore returned the permit immediately while the model load and inference it was bounding continued, so repeated abandoned calls could exceed the configured reranker concurrency and keep several models resident after every caller had returned. (GH#1641) - (embeddings, transcription): the same cancelled-permit leak fixed above for the reranker is now
fixed for
embed_texts_asyncand audio/video transcription. Both bound aspawn_blockinginference call with a concurrency permit held by the awaiting future rather than by the blocking task, so a caller that dropped its future released the permit while the ONNX embedding batch or Whisper inference it was bounding kept running. Transcription is timeout-wrapped by default (transcription.timeout_ms), so an expiry there was a live, designed-in path to the same concurrency breach, not a hypothetical one. (GH#1641) - (pdf): a wrapped numbered heading set in a heading-size font no longer loses its second line.
follows_section's two continuation exemptions only recognized a wrap via a hanging indent or via matching right edges, both of which a no-indent heading's short last line fails by construction. A third exemption,heading_continuation_at_margin, now recognizes a wrap by measuring the heading's own line against the width of the body column beneath it, the same signal the paragraph-merge pass already uses for a body-sized heading. Independently, the paragraph-gap detector could still cut a continuation these exemptions accepted, because a heading's own line pitch legitimately exceeds the body's and was read as a blank-line break; the gap check is now suppressed at exactly the boundary a continuation exemption already accepted. (GH#1650) - (packaging): the Helm chart is published again. The chart publish was gated such that a failed
Docker build leg skipped it entirely, while the container images themselves published and the
release still reported success. No chart was published for 1.1.4, 1.1.5, 1.2.0, 1.2.1 or 1.2.2, so
helm install --version <current>could not resolve. The chart for 1.2.2 has been published retroactively, and a release now fails loudly if its chart is missing rather than shipping without one. - (node): a JavaScript plugin registered through the Node binding now actually runs. Every
generated Node trait bridge --
OcrBackend,PostProcessor,Validator,Renderer,TokenizerBackendand the rest -- was structurally unable to call back into JavaScript. It invoked the JS callable directly from whatever Tokio worker thread the pipeline happened to be on, with no N-API handle scope; it never awaited a returned promise, so anasyncmethod's result was the string"[object Promise]"; and it coerced whatever came back to a string before parsing it as JSON, which fails for a plain object as well. The published.d.tshad always declared the working contract (processImage(imageBytes: Uint8Array): Promise<ExtractedDocument>), so no plugin could have been written against the shipped behaviour. Bridges now hold aThreadsafeFunctioncreated on the JS thread, await the reply throughcall_async_catch(a JS throw surfaces as a plugin error instead of aborting the process), and decode the value without the string round-trip. A handle reference that would otherwise be released off its owning JS thread is leaked rather than freed, since freeing it there corrupts the V8 heap. (GH#1636)
-
(pdf): a page whose MediaBox is not rooted at the origin no longer reports the wrong page size. The native engine's per-page text result took its width and height from the MediaBox upper-right corner instead of its extent, so a page with MediaBox
[10 -100 622 692]reported 622x692 rather than the true 612x792. That figure feeds hierarchy reading-order computation, so a non-zero MediaBox origin could mis-order content, not merely mis-scale it. Pages rooted at(0, 0)— the overwhelming majority — are unaffected, since the two expressions agree there. (GH#1653) -
Upgraded the
crawlbergdependency to 1.7.0 (from 1.6.4). xberg re-exports crawlberg's crawl types --CrawlConfig,MapResult,SitemapUrl-- so the pin is part of this crate's public surface. 1.7.0 makes no Rust API changes; its breaking changes are confined to crawlberg's own Java and Swift bindings, which xberg does not consume.
- (pdf): a two-column page is no longer split down the middle of a column because one line
crosses its gutter. The corridor search treated whitespace as "no span's bbox", so a single
centred footer, caption, or heading set across both columns closed the page's real gutter for
every line on the page. The per-line median then placed the split inside the left column, every
line it crossed became a band boundary, and the reordered text interleaved the two columns
mid-word — producing spliced tokens such as
activamodelanddemandviating. A split that six or more lines run through now also considers corridors at most one line crosses, still bounded by the same distance cap and still refused when either side does not read as a column. Measured over 490 corpus PDFs: 7 documents change, all of them repairing cross-column splices, and none loses vocabulary. Reported and fixed by Data Polder. (GH#1619) - (pdf): a wrapped numbered heading now closes above the run-in beneath it. The GH#1634 fix's closing term also required a right-edge test that only applies to the un-indented case. A hanging indent heading's wrap and a run-in sub-heading below it are both short lines by definition, so their right edges land within tolerance of each other by coincidence, which overrode the correct left-edge answer and held the heading open across the run-in and the body under it. (GH#1637)
- (ppt): a legacy
.pptslide's title is read from the file rather than guessed. The title came from the first line of the slide's merged text, gated on a second line existing and the first being at most 80 characters — so a title-only slide had no title, a two-line title was truncated to its first line, and a longer title was dropped. The outline collection's ownTextHeaderAtomstates which text is the title; that is now read, with the first-line rule kept only for decks that state no title. (GH#1635) - (ppt): the live save's slides are read, not the oldest save's. The persist chain resolved the
current revision correctly and then scanned the stream for the first
SlideListWithText, which on a deck saved more than once belongs to the earliest save — so edited titles and outline text reverted to a superseded revision. (GH#1639) - (ppt): speaker notes attach to the slide that owns them, and the notes master is no longer one.
The notes master's placeholder text ("Click to edit Master text styles…") was emitted as the first
slide's speaker note, and the remaining notes were zipped to slides by position, so every note
after a slide without one landed on the wrong slide. Notes are now keyed by the
NotesAtom's ownslideIdRef. (GH#1640) - (release): the
xberg-cliarchives are attached again. v1.2.0 published 12 assets and v1.2.1 published 54, both with none of the ninexberg-cli-*archives, somise,cargo binstalland every direct download had nothing to fetch. One matrix leg (the aarch64-musl link) failed, and the upload job required all three CLI build jobs to succeed, so a single leg zeroed the whole asset class — as a skipped job, which reports no failure and left both releases looking green. The upload now attaches whatever built, and a following step fails the release when any of the nine is missing. (GH#1638) - (pdf render):
s(close-and-stroke) is no longer dropped, and no longer floods the next fill. The object-level content parser had arms for every path-painting operator in ISO 32000-1 Table 60 excepts, which fell through to an operator every consumer ignores. Two things followed: the stroke was never painted, and — because only a painting arm resets the path builder — the abandoned geometry stayed in the builder and was painted by the next fill. A stroked frame followed by an ordinary white label fill therefore flooded the frame's whole interior, wiping anything already drawn inside it. Text extraction was unaffected: the byte-level fast path always decomposedscorrectly. (GH#1633) - (pdf render): a glyph is no longer discarded because its inferred Unicode is whitespace.
The byte-indexed rasterizer decided whether to paint by asking what character a code represents
rather than whether there was an outline to draw. For a simple TrueType font with a byte-indexed
cmap and no
/Encodingthat character is the glyph id reverse-mapped through the font's own(1, 0)subtable and read as ASCII or Mac Roman — private glyph ordering decoded as an encoding it never expressed. Six byte values were affected (0x09–0x0D,0x20, and0xCA→ U+00A0); re-indexed subsets numbering their glyphs from0x01upward walked through a whole run unpainted while the advance still applied, leaving gaps that read as spaces. (GH#1632) - (pdf): a hanging-number heading no longer swallows body text indented to its title's edge. A line was treated as a numbered heading's continuation whenever the heading had a hanging indent and the line began at the title's left edge — which is equally the geometry of any layout that indents the whole clause, as contracts, tenders and many installation manuals are set. Wrapping now also requires that the preceding line actually ran out of room. Separately, a heading that had absorbed a genuine wrap could never be closed, so the sub-heading and entire body below were pulled in after it. (GH#1634)
- (cli): the Linux musl CLI binaries are published again.
Since 1.2.0 the
aarch64-unknown-linux-muslCLI build has been killed mid-link, and because the upload job requires every musl leg to succeed, no CLI assets were attached to the 1.2.0 or 1.2.1 releases. The cause was the switch to fat LTO: over theallfeature set the final whole-program link exceeded what the arm64 runner could complete. That build now uses arelease-muslprofile with thin LTO; every other target keeps fat LTO. - (php): setting
outputFormatno longer throws. Every call that passed an output format —ExtractionConfig::from_json(json_encode(["outputFormat" => "markdown"])), and anyextract/extractBatchcarrying a per-input config — raisedinvalid type: string "markdown", expected struct OutputFormat. The generatedXberg\OutputFormatclass carried a derived serde implementation that read and wrote its own storage shape,{"type":"markdown"}, while the core enum is externally tagged and writes a bare string, so no real value ever deserialized.EntityCategoryandPiiCategorycarried the same defect. Introduced in 1.2.0 alongside the flat-class change and shipped in three releases. - (wasm):
metadata.formatfields are readable again.metadata.format.titlereturned''andmetadata.format.sheetCountreturnedNaNfor every document, because the value reached JavaScript as aMaprather than the plain objectindex.d.tsdeclares —serde-wasm-bindgenrenders every serde map that way, and serde's internally tagged enum representation is a map. Property access on aMapisundefined. Introduced in 1.2.0 with theFormatMetadataflatten.
extraction::ppt::PptSlideTextcarries the slide's stated title and notes. The struct gainstitle: Option<String>andnotes: Option<String>, so Rust callers constructing it with an exhaustive struct literal must add the two fields. The type is not exposed through any binding.- Breaking (PHP binding):
OutputFormat,EntityCategoryandPiiCategorynow serialize to the wire their Rust counterparts emit —"markdown"for a fieldless variant and{"custom":"latex"}for one carrying a label — instead of the flat{"type":"markdown"}object.from_jsonstill accepts the object form, so existing input keeps working; output moves. - Breaking (WebAssembly binding):
metadata.format— and any other field carrying a flattened, camelCased enum — is now a plain object rather than aMap. This is the shapeindex.d.tshas always declared; code that worked around the discrepancy with.get()must switch to property access. Fields carrying an enum that is not flattened, such aschunking.sizing, keep theserde-wasm-bindgenbridge and are unchanged. - Breaking (C#, Dart, Go, Java, Kotlin, Python, Swift bindings): an enum variant whose name ends
in a multi-letter acronym followed by a single lowercase letter is spelled correctly now.
StructuredDataType.RdFabecomesRdfa(C#, Dart, Go, Swift), Kotlin'sR_D_FAbecomesRDFAandPADDLE_O_C_RbecomesPADDLE_OCR, and Python's stub and runtime agree onRDFA. Wire values are unchanged — only the identifiers move. Names without that shape (IOError,HTMLParser,JSONLD) are unaffected.
- (swift): externally tagged enums now decode the wire the core types actually emit. Regenerated
on alef 0.87.1.
EntityCategory,PiiCategoryandConfidenceSemanticsrelied on Swift's synthesizedCodable, which keys every variant ({"person":{}}), while serde writes a bare string for a fieldless variant and a single-keyed object whose value is the payload ({"custom":"foo"}). Every bridge-constructed value carrying one of these threwDecodingError.typeMismatch—EntityandPiiEntitydecode theircategoryinto a non-optional field, so any result containing an entity failed.OutputFormatis handled separately: itsCustom(String)variant carries#[serde(untagged)], so it round-trips as a bare string rather than a keyed object.
- Dependencies upgraded across the workspace, including
crawlberg1.6.1 → 1.6.3 andtree-sitter-language-pack1.19.0 → 1.19.1.skrifastays pinned at 0.46: 0.47 moves to read-fonts 0.44 while harfrust 0.13.3 is still on 0.43, and the PDF text rasterizer passes a harfrustFontRefto skrifa'sOutlineFace.
-
ServerConfiggainedjob_timeout_secs(default 600 seconds, override viaXBERG_JOB_TIMEOUT_SECSorserver.job_timeout_secs) as the configurable fallback timeout forPOST /extract-asyncjobs whose request does not pin downextraction_timeout_secs. A per-requestextraction_timeout_secsstill always overrides it, and an explicitextraction_timeout_secs: nullstill falls back to this server cap rather than running unbounded. Previously this fallback was a hardcoded 300 seconds, inconsistent with the 600 second default used everywhere else. See the Changed section for the source-compatibility impact. -
A long-running process can now release the embedding and reranker models it no longer uses.
embeddings::evict_modelandreranking::evict_model(and the same functions insparse_embeddingsandlate_interaction) drop one model,clear_engine_cachedrops every model in a cache, andxberg::clear_engine_cachesdrops all of them.set_engine_cache_limitbounds the number of resident engines in a cache and drops the least recently used one first. The default stays unbounded, so existing callers see no change (GH#1626).
-
OutputFormat.customin the Python binding always returnedNone, even when the value genuinely was a custom format. The accessor derived a discriminator from the enum's JSON, which works for every tagged representation but not for this variant, whose payload is untagged and so carries no discriminator to find. It now matches on the variant directly and returns the label. -
The Python type stub declared a
type: strattribute on 23 enum classes that have no such attribute at runtime. The stub emitted it for every data enum, while the runtime only exposes it for enums carrying an explicit serde tag -- so for externally tagged enums (EntityCategory,PiiCategory,OutputFormat) a type checker acceptedcategory.type, which raisedAttributeErroron use. Stub-only change; no runtime behaviour moved. -
The Ruby binding's
FormatMetadata.from_hashandDiffLine.from_hashread a_0key that no longer exists on the wire, so every variant deserialized with anilpayload. All 24 call sites are corrected: the 21FormatMetadatavariants now build their payload from the flattened hash, and the 3DiffLinevariants read thetextkey the enum's#[serde(content = "text")]actually emits (#1594). -
The Python type stub declared format-metadata payloads as
_0(for example_0: ExcelMetadata), a key present neither on the wire nor on the runtime object -- the runtime per-variant getters such as.excelwere always correct. The stub now names the real fields, so type checkers stop reporting valid code as an error. Runtime behaviour is unchanged (#1594). -
The Elixir
Xberg.FormatMetadatatypespec documented ametadata:payload key that matched neither the NIF struct nor the serialized wire. It now matches the struct the NIF actually returns (#1594). -
A Type0 (composite) font's content-stream character codes are now translated to CIDs before glyph widths and vertical metrics are looked up, instead of being used as if they already were CIDs. The two coincide only for
Identity-H/Identity-V, which is presumably why this went unnoticed. A PDF using a non-Identity predefined CMap (UniCNS-UCS2-H,UniJIS-UCS2-H,UniGB-UCS2-H,UniKS-UCS2-H, or their-V/UTF16counterparts) or an embedded/EncodingCMap stream got a wrong width for nearly every glyph -- some over-advancing by up to 4x through the/DWfallback, others under-advancing -- which rendered as stretched or overlapping text and, in extracted text, could split one sentence into a spurious extra paragraph. CIDs now resolve from an embedded/EncodingCMap stream's ownbegincidrange/begincidchardata when present (including a variable-width codespace), else from the font's/CIDSystemInfocharacter collection for the four Unicode-keyed predefined families above, else viaIdentity-H/Identity-Vas before. Measured over the 230-document local PDF corpus: 227 byte-identical -- the expected result, since most PDFs use Identity-H -- and 2 changed, both merging text that a stale glyph position had fragmented. Legacy multi-byte predefined CMaps this crate carries no code-to-CID table for (90ms-RKSJ-H,GBK-EUC-H,B5-H, theUTF8family,UniJIS-UCS2-HW-*,UniJISPro-*, and others) are unchanged -- still wrong, not newly broken -- and now log once per font instead of failing silently (GH#1631). -
A reconstructed PDF table cell now reads left to right instead of in the order its words happened to arrive. The reported symptom was a sub/superscript printing after the rest of the cell --
eta_S %came out aseta % S,Q_HE GJasQ GJ HE-- because a script is drawn as its own content-stream segment a fraction of a point below the line it annotates, so every reading-order sort upstream placed it after the whole line. The same defect also transposed values between columns when two columns were merged into one cell: on a balance sheet whose header reads2017 2016, the row beneath it emitted the 2016 figure first, silently attributing each year's number to the other year. A cell's words are now grouped into visual lines and ordered left to right within each line. Grouping first is load-bearing -- ordering by horizontal position alone interleaves the two halves of a wrapped cell. Each word's own whitespace is also collapsed, so a segment carrying a trailing space no longer stacks it on the separator. Table cells recovered by OCR go through the same ordering (GH#1628). -
Image OCR now honours a PNG's embedded
pHYspixel density instead of assuming 72 DPI. A genuine 300-DPI PNG submitted withtarget_dpi = 300was resized anyway, because the extractor decoded, resized and re-encoded the image -- discarding the density chunk -- before the OCR backend, and therefore before the existingocr.backend_options["source_dpi"]override, ever saw it. Embedded density is now resolved at the extractor boundary and at the backend from one shared implementation, with the explicit override still taking precedence over it. An image carrying no density metadata still defaults to 72 DPI and still resizes (GH#1630). -
A body paragraph is no longer deleted for repeating text that appears elsewhere on the same page. The second
strip_repeating_textpass keyed on lowercased paragraph text with no check that a table was involved, so a sentence matching an earlier title -- differing only in case, with no table on the page at all -- was silently removed. The pass now runs only on pages that have a detected table, removes a paragraph only when that table's own cells carry the same text, and compares case-sensitively. Measured over 230 PDFs: 44 documents changed, 3472 words recovered and 32 lost, both loss cases inspected and benign (one is a restructure whose total content grew, the other two mojibake tokens) (GH#1623). -
OCR text is no longer discarded when a scanned page region is detected as a table but its cell grid cannot be recognised.
recognize_single_tablereturned nothing whenever TATR failed, produced no rows or columns, or the grid failed validation, which threw away every OCR element that had been assigned to that region. A region that cannot be recognised as a table now falls back to emitting its text in reading order, and only when that text is not already carried by one of the page's paragraphs, so nothing is duplicated. Together with the restructuring-heuristic retention guard below, recognised OCR text is no longer silently lost on the layout path (GH#1622). -
extraction_confidenceno longer reports a failed structured extraction as fully schema-valid. The pipeline passedSchemaCompliance::AllValidunconditionally, which is 40% of the combined score under the default weights, so a run whose LLM call failed -- or that was built without theliter-llmfeature, or ran on wasm -- scored exactly as high as one that validated. A requestedstructured_extractionthat leaves nostructured_outputnow scoresAllInvalid. Extractions with nostructured_extractionconfigured are unaffected and keep their previous score;ConfidenceSignalsis unchanged in shape, so no serialized form moves (GH#1624). -
detect_mime_type_from_bytesno longer refuses text that is not valid UTF-8. A byte buffer with no filename or declared type -- a Windows-1252 or ISO-8859-1 CSV export, say -- returnedUnsupportedFormateven though the extractors that would receive it decode legacy encodings throughencoding_rs. Such content is now reported astext/plain, the same answer the UTF-8 path already gave for the same document, so the two encodings of one file behave alike. Content holding a NUL byte, or with too few printable bytes to read as prose, is still rejected (GH#1625). -
The Go binding no longer discards the message of every error the native layer reports. Each known error code was mapped to a typed sentinel (
ErrTimeout,ErrParsing,ErrOcr, and ~20 more) and returned before the message was ever read, so the detail the native layer had already produced -- observed durations, limits, plugin names, counts -- was dropped for all of them; only unrecognised codes kept their text. A timeout surfaced as the sentinel's own placeholder-stripped text,extraction timed out after ms (limit: ms), which reads as a formatting bug but is the whole message the binding ever had, and left callers unable to tell which timeout had fired. The message is now read first and returned alongside the sentinel, soerrors.Is(err, xberg.ErrTimeout)still matches whileerr.Error()carries the real interpolated text. Go was the only binding affected; C#, Java and Zig already read the message before switching on the code. Regression in 1.1.0, when the typed sentinels were introduced. -
The Python package's public option classes regained
from_json.from xberg import ExtractInputresolves to a generated dataclass that shadows the native class at the same name, and that dataclass carried none of the native class's methods, soExtractInput.from_json(...)raisedAttributeErrorwhilexberg._xberg.ExtractInput.from_json(...)worked -- the same name meaning two different things depending on the import. 134 public classes were affected. The dataclasses now delegatefrom_jsonto the native class, so both import paths behave the same. Other native-only methods on those classes (validate,is_empty, thePaddleOcrConfig.with_*builders) are still absent from the dataclass twins and are tracked separately. -
An extraction cancelled by
extraction_timeout_secsnow actually stops its per-page PDF OCR work. The timeout firescancel_token.cancel()at every timeout site, but nothing in the OCR page fan-out read the token, so pages kept being OCR'd after the caller already had itsTimeouterror -- burning CPU and holding OCR concurrency permits, which degrades later extractions in a long-lived process (a server, or anything extracting in a loop). The token is now checked both before spawning a page and inside each spawned task, because the spawn loop finishes almost immediately while tasks queue on the OCR semaphore long after it. A cancelled run also reportsCancelledinstead of tripping the all-pages-failed guard and reporting a wholesale OCR backend failure. -
PDF no longer promotes ordinary body text to a heading. Two gates decide headings independently and neither tested the line's shape, so any line past the title-length floor could be promoted. The sentence-boundary check that should have caught this looked for a literal
". "followed by a capital, but paragraph text joins a block's physical lines with a newline, so every sentence boundary landing at a line end was invisible to the gate while the renderer joined the same lines with a space and displayed it -- the gate and the output disagreed about what the text was. Boundaries are now found across any whitespace, and a line that is mostly bare numerals is treated as a flattened data row rather than a heading. Across 490 documents: 484 unchanged, 5 with fewer headings, 0 with more (GH#1599). -
Hardened the document-global heading/list heuristic's safety check on the scanned-PDF layout-markdown path (
use_layout_for_markdown/ layout detection, force-OCR route). That heuristic rebuilds paragraphs from bare line geometry with no knowledge of the ML layout regions the OCR path already classified, and can silently drop a line its own font-clustering pass treats as furniture or noise; the guard against this only checked that the whole document still had one non-empty element, so a single surviving word anywhere passed it even if an entire page's body vanished. The guard is now a per-restructuring canonical-character retention check against the lossless OCR assembly, and falls back to that lossless assembly whenever any content would otherwise be lost. Compares characters rather than word tokens: a restructuring pass legitimately re-wraps text across the line boundaries it reads (measured case: "list of findings" split across a line came back "list offindings", one dropped space), and a word-token comparison read that benign re-wrap as content loss and rejected legitimate heading/list promotion along with it. This closes a real gap in the guard's own logic; it was not reproduced against a specific "entire page lost" report and should not be read as a confirmed fix for one (GH#1622). -
PDF table/paragraph assembly (
assemble_page_elements_with_tables) now suppresses a paragraph whose words a positioned table's own grid fully carries, so a recognized table no longer also renders its flattened source text as an ordinary paragraph immediately next to the grid -- observed directly (not inferred) on a scanned-PDF fixture with layout detection enabled, where a table's status-row text appeared once as prose and once as a correctly gridded table. Mirrors the GH#1616 precedent from the other direction: suppression requires the table's own cell/markdown content to actually account for every one of the paragraph's words (an order-insensitive multiset match, since a reconstructed grid can reassemble the same words in a different order than the source paragraph), not geometry alone, so a paragraph carrying text the grid does not represent still survives. Note: this closes the duplication for paragraph/table pairs that share a coordinate space (the native-PDF table path). Investigating this also surfaced a separate, unresolved coordinate-space mismatch between OCR/TATR-recognized table bounding boxes and OCR paragraph bounding boxes on the force-OCR + layout-detection route specifically, which currently prevents this same guard from geometrically matching on that route; fixing that is out of scope here and is not yet done (GH#1622). -
PDF no longer deletes text a table's bounding box covers but its grid leaves out. Suppression of text a table already renders was decided on geometry alone, and a reconstructed grid need not span every printed column inside its own bounding box. On a four-column fault-finding grid reconstructed with two columns, every run in the two omitted columns vanished from the document — not in a cell, not in any element, nowhere. A covered run is now suppressed only when the table actually carries its text (GH#1616).
-
PDF no longer cuts a numbered heading that wraps onto a second line. The wrap exemption compared the two lines' right edges, and a wrap's last line is short by definition, so it could never fire: the heading kept only its first line and the rest of its title was emitted as body text. A heading's own continuation is now recognised by its left edge, which is the title's hanging indent rather than the margin body text returns to. Regression in 1.1.5 (GH#1615).
-
An extraction that never requested OCR no longer fails when no OCR backend is registered.
ocr-pipelinecan be enabled without any backend —ocrimpliesocr-pipeline, not the reverse — and in that build the automatic scanned-page trigger aborted an ordinary PDF extraction withOCR backend 'tesseract' not registered. Automatic triggers now check availability and skip with a warning; an explicitforce_ocr,force_ocr_pages,ocr_inline_imagesor caller-suppliedocrconfig still fails loudly (GH#1610). -
Legacy binary
.pptnow reports which slide each embedded picture belongs to. Pictures were read from the OLEPicturesstream, which stores blips in save order and names no slide, so every extracted image carried no page number and every image node was emitted after the last slide. A slide whose only content is a picture therefore produced nothing at all on its own number and read as a blank slide, and captions or any other data keyed on an image's page were filed against the end of the deck. The owning slide is now resolved through the drawing that references the blip; a picture no live shape references is still extracted, without a slide (GH#1620). -
Legacy binary
.pptno longer extracts deleted slide revisions or presents slides in the wrong order. The format is append-only across saves, so editing a deck leaves superseded copies in the stream; treating everySlidecontainer as a slide produced 190 slides for a 96-slide presentation, numbered by byte order. Live slides and their order now come from the persist chain (Current User→UserEditAtom→PersistDirectoryAtom) and the document's slide list, falling back to the previous behaviour if the chain cannot be read in full. Slide numbers are the page every element and chunk of a deck is cited by, so both defects reached consumers as wrong page numbers (GH#1614). -
PDF de-hyphenation no longer welds a compound whose own hyphen falls on a line break. Two sites decide whether a trailing hyphen survives; only one consulted the lexical evidence, so
long-term,cost-effectiveandantigen-presentingcame out aslongterm,costeffectiveandantigenpresenting— tokens that do not exist, and so unreachable by any lexical search. The assembly site now asks the same question the paragraph site already asked, weighing both the static compound list and the witnesses collected from the document itself. A hyphen the wrap genuinely inserted is still removed (GH#1613). -
Legacy binary
.pptno longer loses slide titles. PowerPoint keeps a slide's text in two places, and the extractor read only one: titles held in the document-level outline collection (SlideListWithText) landed in the loose-text bucket, which is discarded whenever any slide exists, so they were absent from the output entirely. Outline text is now attributed to its slide by persist order and merged in, skipping any line the slide's own drawing already carries so a title drawn on the canvas is not duplicated (GH#1612). -
The documented install versions for Java, Kotlin Android, Swift, Zig and the spring-ai integration no longer lag the release. These snippets sit outside
task version:sync, which covers the generated API-reference badges but not hand-authored install directives, so they had been telling users to install 1.1.3 (GH#1593 covers the same class of staleness intest_apps, which is still open). -
OcrConfigno longer rejects valid Tesseract language codes such asfao(Faroese) withInvalid language code 'fao'. Use ISO 639-1 or ISO 639-3 codes.. Config validation checked the language against a general-purpose allowlist that was missing 66 codes Tesseract actually supports, while a separate, Tesseract-specific list already carried them; the two lists had never been reconciled. Config validation itself only started running for configs loaded from files, JSON overrides, or set programmatically in 1.1.0 (previously it ran only in tests), which is when this allowlist gap first became user-visible. Both validators now read from one shared list of Tesseract-supported codes, so this class of divergence cannot recur (GH#1621).
-
Breaking (Java binding): enum constants now follow Java's own convention and are
SCREAMING_SNAKE_CASEinstead of carrying Rust's PascalCase verbatim --LinkStyle.InlinebecomesLinkStyle.INLINE, across roughly 75 generated enums. The JSON wire value is unchanged; only the Java identifier moves, so serialized documents and stored payloads are unaffected. Update references to the constants themselves; generated default values (ChunkType.Unknown,OutputFormat.Plain) moved with the declarations. -
Breaking (Ruby binding): an externally tagged enum variant carrying a single payload (
EntityCategory::Custom,PiiCategory::Custom,OutputFormat::Custom) now serializes the way the Rust core always did --{"custom" => "my-label"}-- instead of wrapping the payload in an extra object keyed by a synthesized positional name,{"custom" => {"_0" => "my-label"}}. Code readinghash[:custom][:_0]should readhash[:custom]. The previous shape matched no other binding and no core output. -
Breaking (PHP binding):
EntityCategory,PiiCategoryandOutputFormatchange from constants-only classes to classes with static factories, because the old shape could not carry a payload at all: a caller-supplied label was silently discarded in both directions, soCustom($label)always round-tripped as an empty string. UseEntityCategory::custom($label)andEntityCategory::person()in place of the oldXberg\EntityCategory::PERSONconstants; the label is readable from the readonly$customproperty. -
Breaking (Node binding): the JSON surface is camelCase throughout, nested types included, and
FormatMetadatais a flat discriminated union keyed byformatTypewhose variant payload fields sit directly on the object ({ formatType: "excel", sheetCount: 2 }). The binding previously exposed two parallel shapes for one Rust type -- an idiomatic camelCase interface beside a snake_case structural twin -- and only the latter was reachable from a result. -
Breaking (Node, Swift bindings):
FormatMetadatanow serializes flat in every binding --{"format_type": "pdf", "page_count": 12, ...}-- matching what the core Rust enum has always serialized (#[serde(tag = "format_type")]), what the OpenAPI discriminator describes, and what the REST API serves. Two bindings disagreed with that wire and have been corrected:-
Node nested the payload one level down under a property named for the variant, so
doc.metadata.format.pdf.pageCountbecomesdoc.metadata.format.page_count. Note the field names are snake_case, unlike the camelCase Node uses elsewhere: the variants carry mutually incompatible field types (headersisstring[]for text and an object array for HTML), so no single Node class can describe them and the value is passed through as serde emits it.formatis typed as a discriminated union inindex.d.ts, so narrowing onformat_typestill gives a fully typed payload. This also resolves the Node and WebAssembly bindings disagreeing with each other -- WASM was already passing serde's shape through, so the two now emit an identicalformatobject for the same document. -
Swift's
FormatMetadatawas atypealiasto an opaque bridge class carrying no payload, so the JSON inMetadata.formatcould not be decoded into anything useful. It is now a realCodable/Sendableenum with one case per format, making the payload reachable:if let json = doc.metadata?.format, case .excel(let meta) = try formatMetadataFromJson(json) { print(meta.sheetCount) }
Metadata.formatstill hands back the serialized JSONString; what changed is thatformatMetadataFromJsonnow yields a pattern-matchable enum carrying the payload instead of an opaque handle. The wire was already correct here -- the Swift type system was the part that was missing.
Python, Go, Java, C#, Kotlin, PHP and Ruby are unaffected on the wire: they either already emitted the flat shape or expose native per-variant accessors over it (#1594).
-
-
Breaking (Rust source, Java):
ServerConfigaddsjob_timeout_secs. Exhaustive Rust struct literals must set the field or use..ServerConfig::default(), and the Java record's canonical constructor gains a sixth component, sonew ServerConfig(host, port, corsOrigins, maxRequestBodyBytes, maxMultipartFieldBytes)no longer compiles -- useServerConfig.builder(), which is unaffected. Every other binding is source-compatible: the field is last and defaulted in the Python dataclass (= 600), the Kotlin data class (= 600L) and C# ({ get; init; } = 600); a defaulted keyword in Ruby and PHP; an optional pointer withomitemptyin Go; and an additivexberg_server_config_job_timeout_secsgetter in the C FFI (gated onapi-types). Deserializing callers are unaffected everywhere -- the field carries#[serde(default)]. -
Public binding-facing structs in this crate are deliberately not
#[non_exhaustive]: alef generatesimpl From<Mirror> for xberg::Twith a struct literal in roughly ten binding crates, and#[non_exhaustive]forbids that cross-crate (E0639) -- including the..Default::default()spread.Defaultplus#[serde(default)]is the forward-compatibility mechanism instead, and a field addition is recorded here as a labelled source break rather than prevented by the type system.#[non_exhaustive]is reserved for types excluded from binding generation. -
Retroactive note for 1.1.4:
Metadata#formatin the Ruby binding changed shape and no changelog entry recorded it at the time. The format-specific payload had been nested under a_0key (format.fetch(:_0).fetch(:title)); since 1.1.4 the payload's fields sit directly alongside theformat_typetag (format.fetch(:title)). Ruby callers written against the older shape raiseKeyErroron_0. The binding has emitted the flat shape since 1.1.4; the generated Ruby e2e specs were still asserting the nested one, which is why this went unnoticed for two releases. Only the Ruby binding is affected. Part of GH#1594, which also tracks the Swift binding still discarding the payload entirely -- that half is not yet fixed.
- The Java binding compiles again. A method returning
Option<Vec<u8>>—Registry.sampleBytesis the only one today — was generated declaringOptional<byte[]>while returning a barebyte[], which javac rejects. 1.1.4 therefore published no Java artifact at all, and the spring-ai integration was blocked waiting on it. Fixed upstream in alef 0.85.12; this release regenerates on it.
- BREAKING (Ruby):
FormatMetadatareaches Ruby as a flat hash. It previously arrived as{format_type:, _0: {...}}, where_0was the name serde invents for an unnamed tuple field; it now arrives as{format_type: 'excel', sheet_count: 2, ...}, the canonical wire the core declares. Code readingmetadata.format[:_0][:sheet_count]must readmetadata.format[:sheet_count]. This shipped unannounced in 1.1.4 and is recorded here retroactively; no other binding's shape changed (GH#1594).
- PDF reading order no longer tears a subscript off the symbol it names. Spans were ordered by the top of their bounding box, but a subscript is drawn 35-40% smaller than its base, so its top sits several points lower even though its baseline is a fraction of a point away. An unrelated span from the next column could sort between a base run and its own subscript, and the symbol the subscript names no longer existed anywhere in the output. Ordering now quantises the baseline into row bands before comparing horizontally, which is what every other caller of that comparator already did (GH#1600).
- PDF table detection no longer bridges two separate tables across the graphics-free gap between them. A cell was built from intersection points alone, so a section heading printed in that gap was absorbed into one of the tables as a single-cell row. A candidate cell now also requires a drawn vertical rule spanning its own Y-range on both sides. The span tolerance is load-bearing: at the tighter X-axis value, rows of a table whose rules are inset by a few points are dropped (GH#1601).
- PDF two-column detection no longer loses the page's split to a hanging-number indent. When any span straddled a correctly detected gutter, the split was replaced outright by the midpoint of the widest whole-page whitespace corridor — on a hanging-number layout, the indent between the numbers and the text. The reorder then hoisted every clause number out of its clause. A relocation is now rejected when it would move the split more than a quarter of the page width, which leaves every legitimate corridor move in the corpus intact (GH#1603).
- PDF paragraph grouping no longer splits a numbered heading that wraps onto a shorter second line. The wrap exemption compared the two lines' right edges, but a heading fills its column on its FIRST line and the continuation is whatever is left over, so the metric was anti-correlated with the answer. The pair is now also exempt when the continuation opens lowercase AND the heading line reaches within a tolerance of the width of what would be merged onto it — the "fills its column" half the original rule stated but never measured. The lowercase test alone is not sufficient: body prose beginning lowercase under a complete numbered heading has the same signature (GH#1605).
- PDF paragraph grouping now recognises a numbered heading whose line arrives as more than one text
span. The break terms tested the predicate against a single span, so a heading set with a hanging
section number —
3.1.7in one span, its title in the next, on one baseline — never looked like a numbered heading and was left to the ordinary paragraph-gap rule. That rule needs a gap wider than ordinary line pitch, so every such heading whose body starts on the next line was welded into it. The line's spans are now re-joined before the predicate runs, which is what the continuation-merge pass already did (GH#1609). - PDF paragraph grouping now recognises a heading whose number is not its first token —
ARTIKEL 1.,Chapter 1,Appendix 1,Annex III,Exhibit A. The numbered-heading predicate is the only boundary signal available when a heading shares font, size, weight and leading with its neighbour, so a heading it could not see was welded onto the line above it, and a run of such headings collapsed into a single element. Recognition is by shape, not by a keyword list: one capitalised word standing in front of an enumerator. Prose that opens the same way —Artikel 12 van de wet is van toepassing.— stays prose, because behind a keyword the text after the enumerator must still be capitalised (GH#1608). - PDF heading detection no longer skips a numbered heading that is only two words long. Promotion
of a bold, body-size line to a heading required more than two words — a floor that keeps short
bold fragments out — and a numbered section title such as
3. PRIJZENor1. INTRODUCTIONfalls below it. Those lines stayed plain bold paragraphs, and a run of them was then coalesced into a single bold line in the rendered output, while the element stream still reported them separately. A numbered section heading is now exempt from the word-count floor; everything else still has to clear it (GH#1611). - OCR no longer adopts a markdown table rebuild that loses content. The rebuilt page replaced the original whenever it was merely non-empty, so a rebuild that dropped text still won. The rebuild is now rejected, with a warning naming both word counts, when it retains fewer words than the content it would replace (GH#1599).
- PaddleOCR's default
model_tierofmobilenow resolves to the pp-ocrv6smalldetection model (9.9 MB) rather thanmedium(62 MB). A tier namedmobilesilently loading the largest available model made a 21-page document take over ten minutes.smallandmediumshare the same 18,708-character dictionary, so recognition coverage is unchanged. The documented model sizes were also wrong and have been corrected (GH#1602). - The PHP extension now loads on Debian 12 and other distributions built against GCC 12. The Linux
publish runners ship GCC 13+, and the extension picked up a
GLIBCXX_3.4.31symbol from their libstdc++ while Debian 12 provides at mostGLIBCXX_3.4.30. libstdc++ is now linked statically; the highest glibc requirement was already below Debian 12's (GH#1606).
Table.cell_stylesandGridCell.heading_level/GridCell.style_nameexpose the paragraph style a DOCX table cell carries. A heading styledHeading1..Heading6inside aw:tc— the banner row forms, questionnaires and datasheets use as a section title, and what Word's navigation pane and aTOCfield treat as the document outline — previously reached every consumer as anonymous cell text. Cell text is deliberately unchanged: prefixing it with#would put a markdown heading inside a table cell. The style travels beside the text instead, so a caller can decide whether aheading 2in a banner row is a section title or a column label.cell_stylesis sparse and omitted entirely for tables whose cells carry no style, so ordinary tables serialise exactly as before (GH#1587).
-
PDF text repair no longer welds two complete words into one.
repair_ligature_spacesremoves the space in…f++i|l|f…to undo a real artefact — some PDFs decompose a ligature glyph and leave a spurious gap, sofirstarrives asf irst— but the same character pattern is an ordinary word boundary whenever a word ends infand the next begins withi,lorf. The only guard was a hard-coded list of 33 short English words tested against the left token, so everything outside it welded, English included:relief forbecamereliefforanditself infringesbecameitselfinfringes. The space is now kept when either fragment is independently attested as a standalone word elsewhere in the same document, reusing the witness mechanism dehyphenation already applies. A fragment appearing only as one half of a candidate pair does not witness itself (GH#1591). -
DOCX page counting no longer collapses a table onto one page. Word writes
<w:lastRenderedPageBreak/>into every cell of a row that straddles a page boundary — one physical break, one marker per cell — and the duplicated markers were reduced to a single break, losing the originals with the duplicates. A seven-page document reported two. Breaks are now identified by table depth, row and cell, so a marker echoed across the cells of one row counts once while several breaks inside a single deep cell each still count (GH#1592). -
PDF outline (bookmark) named destinations now resolve when the
/Names->/Destsname-tree key is UTF-16BE-with-BOM, the form Adobe Distiller writes. The lookup previously decoded the/Destbyte string with a lossy UTF-8 conversion before searching the tree; a name-tree key is a byte string compared by byte (ISO 32000-1 §7.9.6), not text, so the BOM was mangled into replacement characters and every such destination silently resolved toNone, leaving the bookmark'sdestas an unresolvedDestination::Namedwith no page (GH#1589). -
MimeDetectionPolicy::ContentOnlyno longer rejects a legacy OLE2 Office document (.doc/.xls/.ppt) passed by path when the same bytes are accepted through the bytes API. Path-based content detection only sniffs the first 4 KB of a file, but an MS-CFB compound document cannot be typed from a prefix — identifying it means following the FAT sector chain to the root directory entry, which a truncated buffer cannot do. Detection now falls back to a structure-aware read of the file for a compound-file header that a 4 KB prefix left inconclusive, the same escape hatch a ZIP-based Office document already had for the same class of failure (GH#1590). -
PDF table detection no longer invents a column boundary from a rule that stops short of the row band.
BAND_RULE_SPAN_TOLwas defined asSNAP_TOL, conflating two different questions:SNAP_TOLdecides whether two coordinates are the same coordinate, while this one decides whether a vertical rule runs through a band. At 3pt an edge could fall short at each end and still count as spanning, so a band up to 6pt shorter than the rule beside it was cut where the drawn rule gave it no boundary. Those phantom columns are what let a band of prose inside a drawn frame split into cells and qualify as a table, which on the reported document cost page text. Now 1.0 and deliberately independent ofSNAP_TOL(GH#1588). -
Tesseract
psm = 0is now rejected at configuration validation. PSM 0 is Tesseract'sPSM_OSD_ONLY— orientation and script detection with no character recognition — so it cannot satisfy a text-extraction request, and Tesseract emits no hOCR for it at all. Setting it previously succeeded while returning either a zero-length document or degraded, partially dropped text, depending on the Tesseract build, in both cases with no warning and at several times the cost of a normal run. The error now names the mode and points at 3 (auto), 6 (single block), and 11 (sparse text). Valid values are 1-13; omittingpsmcontinues to let the pipeline choose (GH#1586).
This release contains a breaking public API change.
TesseractConfig.psmis now optional. Callers that read or set it as a plain integer must handleNone/null— see below.
- Breaking:
TesseractConfig.psmis nowOption<i32>(null/None/absent in the bindings) and defaults to unset rather than to 3. This fixes supplying aTesseractConfigat all acting as a hidden behaviour switch: because several code paths keyed on the struct being absent, a caller who set one unrelated field — table detection, a preprocessing knob — silently lost the whole-image PSM 11, the vertical-language PSM 5, the layout-region PSM 6, and the sparse-text retry, and got Tesseract's PSM 3 instead.TesseractConfig()with default fields is now a no-op: the pipeline applies exactly the same automatic PSM it would with noTesseractConfig. An explicitly setpsmis still honoured. Bindings that modelpsmas a plain integer expose a companion presence check (for examplexberg_tesseract_config_has_psmin the C API), since a bare integer cannot distinguish "unset" from a real0.
- Fixed a numbered or bulleted list on a scanned page being reconstructed as a table, replacing
the list text with a mangled grid. A candidate region whose first column is list markers
(
1.,a),•) end to end — the header cell included — is now rejected on the OCR routes. A genuine numbered table is unaffected: its first column carries a header label (Line,Item) above the numbers, which is what separates the two. - Fixed a table detected on a scanned page having its text returned twice — once as paragraphs,
once as table cells — in the document content and element tree. This affected every
output_format;"plain"only appeared to avoid it. - Fixed
PdfConfig.top_margin_fraction/bottom_margin_fractiondefaulting to 0.06/0.05 (6%/5%) since 1.1.0, which silently dropped OCR text — page titles included — in the top and bottom bands of every default-config scanned PDF page with no warning. Both now default to 0.0 (disabled); set them explicitly to filter header/footer content. The nonzero defaults also forced every default-config OCR page onto the lossy per-page route instead of a document-capable backend's whole-document path; that routing is restored too. - Fixed rendered PDF pages losing the Tesseract backend's own
ProcessingWarnings (including the dictionary-filter removal notice) and OCR metadata (psm,language,tesseract_dict_invalid_word_ratio), both of which reached the caller for a standalone image but were silently dropped for the same page rendered from a PDF. - Fixed
OcrConfig.languagebeing discarded whenever aTesseractConfigwas supplied, so a German document was OCR'd in English. One precedence rule now governs both Tesseract backends and the vertical-language check. - Fixed supplying any
ImageExtractionConfigsuppressing document-level OCR on scanned PDFs, which returned empty pages with only a debug log. - Fixed rendered PDF pages ignoring the configured render DPI.
target_dpi,min_dpi,max_dpi, andauto_adjust_dpiare now honoured. With no configuration the default stays at 150 DPI, unchanged. - Fixed suspended hyphens being welded during text assembly, turning
onderhouds- enintoonderhoudsen. A hyphen is now joined only across a genuine visual line break, matching the rule the pipeline layer already applied. - Fixed an unruled full-width band in a ruled table being cut at column positions no rule gives it, splitting headings mid-word. A column boundary now counts only where a vertical edge actually spans the band.
- Fixed every non-header table cell having its em-dashes, en-dashes and minus signs rewritten to an
ASCII hyphen, the spaces around a hyphen collapsed,
E-/E+lowercased toe-/e+, and any cell consisting solely of a dash emptied. That normalisation is correct for a numeric column (an em-dash means nil,1.5E-05is an exponent,- 3is-3) but corrupted prose tables, turningFunctionaliteit—12intoFunctionaliteit-12and a part codeHRE - HRecointoHRe-HReco. It is now applied only to columns whose data cells are predominantly numeric (#1582). - Fixed the Windows PHP extension archives failing to publish at all.
vendor-windows-native-closure.ps1repacks a.zipwithCompress-Archive, which runs no native command and so never sets$LASTEXITCODE; the release workflow gated on it, and an unset$LASTEXITCODEcompares as non-zero, so every Windows archive was rejected immediately after being vendored successfully. Combined with an all-or-nothing matrix gate that withheld the release's PHP assets whenever any single leg failed, this left v1.1.0 and v1.1.1 with no PHP binaries at all. Both are fixed: the script now sets its exit contract explicitly, matching its sibling scripts, and the upload job now ships the archives from the legs that succeeded (#1585).
This release contains a breaking public API change.
OutputFormat::Structuredis renamed toOutputFormat::DocTags. Update any config, CLI invocation, or binding call usingoutput_format = "structured"to"doctags".
- Breaking: renamed
OutputFormat::StructuredtoOutputFormat::DocTagsacross every binding and theoutput_formatconfig field. The rename shipped in 1.1.0 but was only alluded to there, with no entry describing it; the variant was renamed, not removed, and is available as"doctags". Anoutput_formatof"structured"is not rejected — it resolves to a custom renderer of that name, which is not registered. - Breaking: the TypeScript and WebAssembly
OutputFormattype is a string union again ("plain" | "markdown" | "djot" | "html" | "json" | "doctags" | ...), matching the serde wire format shared with the CLI, REST, MCP, config-file, and Go surfaces. 1.1.0 briefly published an object union ({ type: "markdown" }) for these two bindings only.
- Fixed PHP extension packaging, which produced no PIE archives for 1.1.0.
- Fixed HEIC and AVIF decoding on the Linux (glibc) Node binding, which shipped a
libheifbuilt with no HEVC or AV1 decoder at all — every.heicand.avifinput failed to decode while theheicfeature still reported as present. The Elixirlinux-gnuNIF carries the same working codec closure. - Fixed Elixir NIF publishing for
linux-gnuand Windows, which produced no artifacts for 1.1.0 and left the Hex package at 1.0.14. Thelinux-gnuNIF is now built against the glibc 2.28 floor it claims to support; the artifacts published for 1.0.14 bundled HEIF codec libraries that required a newer glibc. - Fixed the Windows Hex package, which declared the
x86_64-pc-windows-gnutarget while CI built and publishedx86_64-pc-windows-msvc.RustlerPrecompiledresolves the msvc triple on Windows and rejects any triple the package does not declare, somix deps.getfailed with "precompiled NIF is not available for this target" even though the artifact existed. Windows users had to compile the NIF from source.
- The Linux binding images now verify a pinned SHA-256 for every vendored native dependency
(
libde265,libheif, ONNX Runtime) before building it, instead of trusting the download. These libraries are linked into the published Node and Elixir artifacts.
This release contains breaking public API changes. Entries prefixed Breaking: below remove or change public API — notably
OutputFormat::Structured, theElementIdwrapper,ExtractedDocument.formatted_contentin the language bindings, and thecore::batch_mode,core::formats, andcore::iomodules — and configuration deserialization now rejects unknown fields rather than ignoring them. Review them before upgrading.
-
Added per-page OCR confidence to
PageContent.ocr_confidence, reported as aPageOcrConfidence { score, word_count, backend }(#1568). The field is absent for pages that were not OCR'd.scoreis populated only for backends whose confidence is a calibrated legibility scale (normalised to0.0..=1.0) and isNonefor uncalibrated ones, so a page OCR'd without a comparable score is still distinguishable from a page nobody scored. It is reported alongsideword_countbecause noise filtering runs before the score is computed: a high score over very few surviving words does not mean the page read well. -
Added HWPX (Hangul Word Processor XML) extraction to the WebAssembly package.
unhwptarget-gates its ZIP reader to a deflate-only, LZMA-free build underwasm32, so the native-C dependency that previously kepthwpxoffwasm-targetdoes not apply there. -
Added diagram recovery from flat OpenDocument drawings (
.fodg), including content-based detection of theapplication/vnd.oasis.opendocument.graphics-flat-xmlMIME type. Connectors name their endpoints outright, so the recovered graph is exact rather than inferred from geometry (#1545 corpus fixture). -
Added structural extraction for MyST Markdown syntax and MyST text notebooks, including saved inline
{eval}values in Jupyter markdown cells (#1538). -
Added extraction of Jupytext percent- and light-format notebook scripts, including
text/x-python,text/x-r-source, andtext/x-juliaMIME aliases (#1538). -
Added bounded, cancellable SQLite and GeoPackage table extraction with schema-based GeoPackage detection,
.sqlite3and.gpkxfilename support, and defensive handling for untrusted databases (#1510). -
Added configurable MIME inference policies for preferring content signatures, trusting supported filename extensions, or ignoring extensions, with per-input overrides (#1509).
-
Added native KML and GeoJSON extraction with canonical MIME routing (#1508).
-
Added Rust
SUPPORTED_FORMAT_COUNTandSUPPORTED_EXTENSION_COUNTconstants derived from the MIME registry, plus automated synchronization for published format-count claims (#1511). -
Added reusable Rust PDF render sessions for querying page counts and rendering multiple pages without reopening the document (#1485).
-
Added cooperative cancellation for single and batch extraction (#1476).
-
Added dynamic system linking for Tesseract and Leptonica through the
tesseract-dynamicfeature (#1407). -
Added managed Azure AD, Google Vertex AI, and AWS STS credential providers, with credential values redacted from debug output.
-
Added reasoning-effort, provider-specific request-body, and Bedrock configuration for LLM extraction.
-
Added
xberg doctorand the Rustdoctor()API for validating configuration and probing every compiled OCR, VLM, layout, table, formula-recognition, and cache capability without downloading models or contacting remote providers.xberg doctor --cleanremoves stray files only from Xberg-owned caches (#1347). -
Added the Sceptre EasyOCR Gen2 backend for desktop, mobile, and WebAssembly.
-
Added sparse and late-interaction embeddings to chunk output.
-
Added a Prometheus
/metricsendpoint to the API server (#1391). -
Added explicit CSV delimiters and comment-line prefixes through
CsvOptions. -
Added
xberg tree-sittercommands for downloading, listing, and cleaning language assets, with optional configuration-file loading. -
Added VLM extraction for complex PDF regions and LaTeX formula extraction from VLM OCR.
-
Added structural AsciiDoc and WebVTT extraction.
-
Added Docling DocTags input and output, including tables and page geometry (#1383).
-
Added formula recognition for rasterized pages and exposed formulas consistently across extracted formats (#1385).
-
Added JATS, EPUB, ODT, and ODP MathML-to-LaTeX conversion.
-
Added deterministic diagram recovery from SVG and PDF sources with Graphviz DOT output (#579).
-
Added
SecurityLimits.max_pagesfor PDF, presentations, Keynote, and multi-frame TIFF documents (#1451). -
Added explicit PDF backend selection through
PdfConfig.backendand--pdf-backend(#1448). -
Added musllinux Python wheels and a Windows x86_64 Ruby gem.
-
Added PDF and HTML extraction plus layout and transcription types to the WebAssembly package.
-
Added
--ocr-no-cacheto bypass the Tesseract result cache. -
Added
ContentFilterConfig.include_footnotesfor retaining footnotes classified as page furniture. -
Added a public
render_heading_breadcrumbhelper for retrieval-oriented chunk content (#1393). -
Added structured-output merge, citation, and vision-fallback helpers for Rust embedders.
-
Added a Tower-compatible extraction service, request type, and builder for Rust applications.
-
Added typed configuration for TrOCR, PaddleOCR-VL, GLM-OCR, and DeepSeek-OCR backends.
-
Added
classify_chunks_ownedfor classifying and returning an owned document. -
Exposed chunk-classification and LLM concurrency, provider, cache, budget, and rate-limit configuration types at the Rust crate root.
-
Added
OcrConfig::security_limits.ExtractionConfig::security_limitsis now threaded through to every OCR route — embedded images, Tesseract, PaddleOCR, and scanned PDF pages — instead of each route decoding images under a hardcodedSecurityLimits::default()(#1554). -
Added
detected_language_confidences, carrying each detected language's confidence, proportion, script, and reliability alongside the existingdetected_languagescodes, so a document that is 95% English and 5% French is distinguishable from an even mix (#261). The existing field keeps its type and ordering. -
DOCX reviewer comments now emit their own
NodeContent::Commentnode instead of riding the footnote reference and definition machinery, so consumers can tell a comment from a footnote. -
PDF annotations now preserve their subtype (Ink, Square, Circle, Polygon, PolyLine, Line, Squiggly, Caret, FileAttachment, Sound, Movie) instead of collapsing to
Other, carry author, modification date, colour, subject, and QuadPoints, recover the text a Highlight marks, and are emitted by the Markdown, Djot, plain, HTML, and JSON renderers — previously no renderer emitted annotations at all (#63). -
PDF extraction now reads image alt text from the structure tree, falls back to XMP for title, author, and subject when the Info dictionary is empty, surfaces
/PageLabels(roman-numeral front matter, per-section numbering) throughmetadata.additional, excludes content on optional-content layers that are off by default, and renders filled AcroForm values. Unencodable images, annotation failures, and form failures now emit aProcessingWarninginstead of being dropped at log level (#62, #71). -
The OOXML
DocSecuritybit field is decoded into named protection flags onMetadata.additionalfor DOCX, XLSX, and PPTX, so a password-protected or read-only-recommended document is distinguishable from an unrestricted one. -
Added PaddleOCR on the tract backend, so classical PaddleOCR (DBNet, CRNN, AngleNet) is available on
wasm32and the Android x86_64 emulator, where ONNX Runtime cannot link. -
Added
top_p,stop,seed,presence_penalty, andfrequency_penaltytoLlmConfig, validated and applied to every outgoing request. They were previously accepted by every config file and language binding and then dropped before reaching a provider. -
Added
LlmConfig.max_concurrencyto bound VLM OCR and image-captioning requests in flight independently ofConcurrencyConfig.max_threads, which represents local CPU capacity (#1453). -
Every error variant now carries a stable FFI error code, so typed error handling works in the C-ABI bindings;
errors.Is(err, ErrOcr)in Go, Java'scheckLastErrorswitch, and Zig's error set previously collapsed all variants to a single unknown constant. -
Exposed
html_to_markdown_rs::ConversionOptionsat the Rust crate root, so callers configuringExtractionConfig::html_optionsno longer need a direct dependency on the upstream crate, and madeDocumentNode's text and node-type accessors public soDocumentStructure.nodescan be read as documented. -
Added
FormatMetadata::html(), returning the HTML metadata when the variant isHtml, matching the accessors already exposed for the other formats. -
Added an opt-in Pdfium PDF extraction backend behind the
pdf-pdfiumfeature, selectable withPdfConfig.backendor--pdf-backend pdfium, providing page count, per-page text, and Info dictionary metadata. Its scope is deliberately narrower than the native engine — no table detection, layout integration, form fields, or OCR fallback — and every result carries aProcessingWarningnaming the gap. The feature is not part offull, so it reaches source builders only. -
Added a Scoop manifest published to the
xberg-io/scoop-bucketon release, so the Windows CLI can be installed withscoop install xberg. -
Extraction now reports a
ProcessingWarningwhen a document decodes lossily or degrades silently. Decode provenance is captured before mojibake cleanup strips the replacement characters that used to be the only evidence, and archive, AsciiDoc, WebVTT, XML, and plain-text extraction warn on replaced characters. Unresolved ODT image hrefs, unparseablestyles.xml, collapsed repeated table cells, skipped LaTeX, Typst, RST, and Org includes, OPML without a body, links past the per-document URI cap, truncated XML, and words Tesseract failed to extract now warn instead of failing silently (#171, #133). -
A PDF page whose raster render comes back blank now falls back to OCR'ing the page's embedded image XObjects, and that recovery preserves the tables, formulas, LLM usage records, and image preprocessing metadata the backend produced instead of keeping only the text, with every recovered payload accounted against
security_limits.
-
Breaking (Python binding):
ExtractionConfigandDoctorReportare now frozen dataclasses rather thanTypedDicts, matching the 121 option types that were already dataclasses. Passing a plaindictor a JSON string asconfigstill works —extract()coerces both — but anExtractionConfigobject no longer supports mapping operations, soconfig.get("chunking")andconfig["chunking"] = ...now raiseAttributeError/TypeError, and the instance is immutable. Build a modified config withdataclasses.replace(config, chunking=...). -
PDF parsing no longer reports recoverable input at WARN. A missing embedded font, an object outside the xref table, an unreadable CFF version, and a reading-order fallback are ordinary properties of real PDFs rather than conditions an operator can act on; they are now TRACE (or DEBUG for strategy fallbacks), and each document emits a single DEBUG summary on the
xberg_native_pdf::recoverytarget carrying the totals instead of one event per occurrence. Measured over a 4,000-document corpus this removed 4,012,488 of 4,014,206 log events, against which 44 genuine parse failures had been sitting at a ratio of about 1 in 91,000. ERROR behaviour is unchanged — it already corresponded one to one with documents that failed (#1547). -
Breaking (Rust source):
validate_mime_typeno longer accepts any value with animage/prefix. It now parses the MIME type and requires exact membership in the supported-format registry, so unregistered vendor image subtypes such asimage/x-custom-formatare rejected asUnsupportedFormatinstead of validating (#1511). -
Per-page OCR recognition-noise detail (fragmented-word ratio, word count, mean confidence) now reaches the page accept/reject decision and is emitted at
DEBUGinstead of being discarded one frame earlier. No threshold is gated on it yet; the blended stage score alone cannot discriminate noise pages. -
Breaking (Rust source):
ExtractionConfigaddsapply_notebook_cell_tags. Notebook extraction now honors MyST and Jupyter Book remove/hide cell tags by default; set the field tofalseto retain all saved cell content (#1538). -
Breaking (Rust source):
OcrQualityThresholdsaddsdiscard_suspected_ocr_noise; exhaustive struct literals must set the field or use..Default::default(). -
Breaking: configuration deserialization now rejects unknown fields in nested Xberg configuration tables instead of silently ignoring misspelled settings.
-
Breaking: PDF backend configuration now uses
"native"andPdfBackend::Nativeinstead of"pdf_oxide"andPdfBackend::PdfOxide. Update explicit configuration values; the default is unchanged. -
Breaking:
EmbeddingModelType::LlmandRerankerModelType::Llmnow carry their model name in the enum variant. -
Breaking:
Formula.bboxandFormula.pageare optional so formulas from formats without page geometry can be represented. -
Breaking: unknown multipart fields on extraction endpoints now return an error instead of being ignored.
-
Chunk
contentnow contains the exact source span; heading breadcrumbs are available separately. -
The CLI
allfeature now includes audio transcription. -
security_limits.max_pagesnow applies to presentations, Keynote, and multi-frame TIFF as well as PDF. -
create_client_with_credential_providernow returnsManagedClient, and an LLM concurrency limit of zero is rejected. -
Native PDF pages now expose their final per-page reading order.
-
WebVTT cue timing is optional for blocks without a timing line.
-
OpenDocument packages without
content.xmlnow return an extraction error. -
CLI text output now includes the extraction envelope with warnings, timings, and metadata.
-
CLI JSON output now reports peak resident memory.
-
Windows builds now include the same supported feature set as other desktop builds.
-
Breaking: Rust element identifiers now use
Stringdirectly; theElementIdwrapper has been removed. -
Breaking: Public tuple fields for ranges, coordinates, dimensions, links, code blocks, and attributes now use named Rust structs and serialize as JSON objects. Legacy positional JSON arrays are still accepted when parsing, so payloads written by 1.0.x keep deserializing, but they are no longer emitted.
-
Breaking: removed the duplicate
xberg::llm::region_extractor::RegionKind; importxberg::RegionKindinstead. -
Parsing and configuration deserialization now reject invalid region, redaction, and reranker values.
-
Corrected and expanded installation, CLI, configuration, extraction, migration, integration, and cross-language API documentation.
-
Corrected canonical MIME and extension routing for DBF, YAML, reStructuredText, Org, Typst, XHTML, Djot, JPEG 2000, HEIC/HEIF, MP4, and MPEG inputs.
-
GeoJSON extraction now returns a bounded aggregate summary by default, including feature, geometry, property-key, position, and bounds metadata. Set
geojson.include_full_coordinates = trueto retain the complete document and coordinate arrays. -
quality_scorenow explicitly measures the cleanliness and readability of retained text, not extraction completeness; inspectprocessing_warningsfor known partial or degraded results. -
The default
security_limits.max_table_cellsremains 100,000 aggregate cells per document; limit errors now explain how to raise it for trusted inputs or reduce the source table. -
TesseractConfig.language_model_ngram_onnow defaults totrueon both the PDF and standalone image OCR paths. Tesseract previously applied no penalty to output that does not look like a word of the target language, the dominant failure mode on scanned line art. Set the field tofalseto restore the previous behaviour. -
Tesseract Markdown-format OCR now drops hOCR lines whose dictionary-checkable words are more than 60% invalid, removing recognition noise such as
OWATS DNDEVETwhile keeping labels likeEXHIBITandLEGEND. A line needs at least two checkable words to be scored, and the removed-line count is reported as aProcessingWarning. -
Undecodable-text OCR routing is now decided per page rather than for the whole document, so a single unreadable page no longer sends every page of a PDF through OCR and discards good native text. The previous document-wide fallback still applies when page boundaries are unavailable or inconsistent.
-
With
max_threadsunset the thread budget ismin(num_cpus, 8)and now ceilings Rayon, ONNX Runtime intra-op threads, and batch workers alike. A cgroup CPU quota is honoured in place of the hardcoded 8 where one exists, and a host with more than 8 cores and nomax_threadsis warned once per process (#1392). -
PaddleOCR inference now uses the resolved thread budget instead of a hardcoded single thread. The session is serialized behind a mutex, so exactly one worker runs and can claim the whole budget without oversubscribing.
- Breaking: removed the inert
ChunkingConfig::prepend_heading_context,breadcrumb_target,BreadcrumbTarget, and corresponding CLI and environment options; use chunk metadata orrender_heading_breadcrumbwhen a retrieval index needs headings inline. - Breaking: removed
OutputFormat::Structured; usePlainfor unrendered content orJsonfor a structured content tree. - Breaking: removed
ExtractedDocument.formatted_contentfrom language bindings; usecontentor select the desired output format during extraction. - Removed advertised support for troff, mdoc, POD, and DokuWiki because they did not have structural extractors.
- Removed fabricated OCR
script_nameandscript_confidencevalues. - Removed the unused public
LanguageRegistry,BatchProcessor, object-pooling APIs, and unused tree-sitter re-exports. - Removed the nonfunctional
wasm-threadsfeature. - Removed PDF writing, editing, building, and XFA conversion APIs from the native PDF crate; read-only XFA analysis remains available.
- Breaking: removed the inert
Enginestructured-policy, preset-resolver, LLM-client, and model-provider injection methods. - Breaking: removed the inert transcription field from
EnrichmentConfig; configure transcription during extraction instead. - Breaking: embedding, reranking, sparse-embedding, late-interaction, and preset APIs are now exposed only when their required features are enabled.
- Breaking:
core::batch_mode,core::formats, andcore::ioare now crate-private, and the publicDocumentStructureBuilderhas been removed.
-
Fixed the Windows Ruby gem failing to build.
xberg-libwpd's build script chose its zlib by operating system alone, so the gem's MinGW/UCRT toolchain was handed vcpkg's MSVC-builtx64-windows-static-mdarchive and the link died withcorrupt .drectve/ld returned 5. The vcpkg path is now taken only for genuinely MSVC targets; every other target links the static zliblibz-sysalready builds from source. -
Fixed
XbergLoaderignoring chunking and per-page splitting whenever the LangChain integration was given anExtractionConfigobject. Both settings were read only when the config was adict, so afterExtractionConfigbecame a frozen dataclass the documentedExtractionConfig(pages=PageConfig(extract_pages=True))andchunking=ChunkingConfig(...)forms silently produced one Document per file instead of one per page or chunk. The config is now read as an object or a mapping. -
Fixed a ruled troubleshooting page collapsing into one table, taking its section headings down with it as cell text.
split_rows_by_text_positionssubdivides a producer-drawn row band by the Y positions of the text inside it, and since the #1555 fix a candidate split was accepted only when EVERY resulting Y-cluster carried text in at least two columns, with the rejection all-or-nothing for the band. A band that mixes multi-column data rows with single-column lines -- a section heading, a lead-in, a wrapped continuation -- can never satisfy that, so one such line vetoed the split for the whole band and every line inside it became cell text. On one 56-page installation manual, six ~20 pt row bands became a single 522 pt table, the document went from 808 elements to 759, and four numbered headings disappeared from the outline. The band is now split once at least two of its clusters are independently evidenced, and each deficient cluster is resolved on its own terms: it folds into the cluster above only when it introduces no column that cluster left empty, which is the signature of a wrapped continuation. Anything else -- a heading, a lead-in -- stays a row of its own, one cell wide, which is what such a line inside a ruled band actually is. Two independently evidenced clusters are required rather than one because a single evidenced cluster can be coincidence, which is precisely the #1555 case (#1565). -
Fixed a word split across two touching PDF spans being rejoined with a space, so
prijsextracted aspri js. The gap between the two spans measures 0.069 pt -- 0.008 em at 9 pt, against a 2.5 pt space glyph -- on an identical baseline at an identical font size, so no gap threshold produced the space:segments_need_spacereached one of its unconditionalreturn truebranches first.SegmentDatakeeps onlyis_bold/is_italic/is_monospaceand dropsfont_name, so a mid-word switch between two embedded subset fonts whose/FontDescriptors disagree onForceBold,ItalicAngleorFixedPitchreads as a style change carrying no geometric signal at all. That is why the defect never reproduced against base-14 Helvetica, and why widening the gap to 2 pt changed nothing. A touching-spans guard now runs before those branches: two segments on the same baseline, at the same font size, with alphanumeric characters on both sides of the boundary and a gap under 0.025 em are one word and are concatenated. The guard can only join, never split, and it never fires across an explicitly drawn space. The table path needed the same test one stage earlier, insegments_to_words, becauseHocrWordis integer-rounded and cannot represent a sub-point gap by the time cell text is joined. Affects ordinary prose, not just tables: of 18 confirmed cases, 14 wereNarrativeText, 3ListItemand 3Table(#1566). -
Fixed PDF table reconstruction dropping early rows when data-start inference classified more than two leading rows as headers. The two-row header cap is retained, but surplus inferred header rows are now demoted to data in source order instead of being discarded (#1558).
-
Fixed native PDF top-to-bottom reading order splitting one visual table row at an absolute 3-point coordinate-band boundary, which could move an article number before its position and fuse the two identifiers. Visual rows now use an anchored, font-scaled tolerance, reconstructed lines restore left-to-right fragment order, and narrative assembly preserves a separator after a severe geometric backtrack (#1560).
-
Fixed PDF dehyphenation treating inline run/style boundaries as visual line wraps. Suspended hyphens such as
vracht- en verzendkostenare now preserved, while compounds genuinely split across different baselines are still rejoined (#1561). -
Fixed DOCX page attribution staying permanently low after Word omitted a rendered-page marker between vertically stacked inline images. The parser now conservatively infers missing breaks from each section's usable page height, including documents with different section geometries (#1559).
-
Fixed DOCX DrawingML and VML text boxes dropping XML and numeric character references such as
&and€from extracted text (#1562). -
Fixed OCR image decoding ignoring the caller's configured
security_limits. Every OCR route — embedded images, Tesseract, PaddleOCR, and scanned PDF pages — decoded raw image bytes under a hardcodedSecurityLimits::default(), so raisingExtractionConfig::security_limitsto accept a large scan still had it rejected at the OCR decode step. The configured limits now reach all four routes, and PaddleOCR also honors a per-callbackend_options["security_limits"]override (#1554). -
Fixed a drawn PDF table row with a wrapped cell being shattered into extra rows. Splitting a row band by text Y-position now requires at least two columns to have independent text evidence for every candidate row before splitting; a band where only one column wraps to a second line now stays a single row (#1555).
-
Fixed monospace font detection matching any font name containing "mono", misclassifying foundry names such as "Monotype Corsiva" as a monospace font and skewing the word-spacing heuristic and code-block detection that depend on it. "Monotype" is now excluded from the substring match, and the PDF text run buffer's separate ad hoc monospace check was replaced with the same shared helper.
-
Fixed a standalone multi-line monospace paragraph not being recognized as a code block unless it had a consecutive monospace neighbor paragraph. A lone paragraph that already carries two or more monospace lines is now fenced as a code block on its own (#1557).
-
Fixed PDF text extraction silently corrupting ordinary text. A contextual ligature-repair pass rewrote
:totiand an uppercaseMbetween lowercase letters tottion every element of every document, mangling identifiers, ratios, times, URLs, and units such asnM(for exampleaMbbecameattib). The repair was introduced for European PDFs that encode ligature glyphs at ASCII code points, but it was gated at the time on a per-font broken-CMap signal from pdfium'shas_unicode_map_error(). That gate was lost when pdfium was removed as a backend and was never ported to pdf_oxide, leaving the rewrite running unconditionally. Both substitutions are removed; they can only return alongside a real document-level evidence gate (#1556). -
Fixed optional fields in the Python and PHP bindings rejecting payloads that omit them. The generated mirror structs lost their
#[serde(default)]attributes, so deserializing a document whose JSON left an optional field out failed instead of falling back to the default. -
Fixed legacy
.docheadings being guessed from line length rather than read from the document's own styles. A paragraph styledheading 1..heading 9— directly or through a custom style derived from one, such asTOC Heading— now becomes aHeadingat that level, instead of every detected heading being a level 2. Documents that apply no heading style keep the previous shape-based detection, because roughly half the test corpus styles its headings as boldNormaland would otherwise lose every one; the choice is made per document, not per paragraph. A heading-styled paragraph that is also list-bound stays aListItem, matching how the DOCX path treatsw:numPr(#1553). -
Fixed legacy
.docautomatic list numbering being dropped entirely: a paragraph Word numbers through its list tables arrived as prose, indistinguishable from an unnumbered sentence, while the DOCX path emitted aListItemfor the same construct. Auto-numbered paragraphs now arrive asListItems inside an ordered or bulleted list container, with their nesting depth, matching the DOCX path. The number Word paints (1.1,a.) is still not rendered — recovering it needs list-table counter state — so a document mixing automatic and hand-typed numbering shows the typed numbers as text and the automatic ones as list structure (#1550). -
Fixed legacy
.docelements being split on blank lines rather than on Word's paragraph marks, which merged every pair of consecutive paragraphs not separated by a blank line into a single element. One corpus letter returned its entire ten-paragraph body as one element. Word97 and later documents now emit one element per Word paragraph, matching what the DOCX path does withw:p. This changes element boundaries, counts and indices for most.docdocuments, and alterscontentline spacing accordingly; consumers keying on element position will see the difference. Word 6/95 documents and those falling back to contiguous text extraction keep the previous blank-line behaviour, because they carry no paragraph properties to use. -
Fixed legacy
.docextraction readingfcClxfromFibRgFcLcb97pair 66 — an obsolete field Word writes as zero — instead of pair 33, so the piece table was never walked for any document and extraction always fell back to readingreserved5/reserved6, bytes [MS-DOC] requires a reader to ignore. Where those bytes disagreed with the real text start, whole documents were decoded as UTF-16LE and returned as glued CJK-looking code points; multi-piece and fast-saved documents could not be assembled at all. Footnote, header/footer, comment, and text-box subdocument text now also reaches the output for these files (#1551). -
Fixed the Elixir NIF's vendored
Cargo.lock, shipped in the Hex package, pinningtree-sitter-language-pack1.15.12 while the crate requires 1.16.1 — a source build of the NIF with--lockedcould not resolve. This affects anyone whose platform has no precompiled artifact and therefore builds from source. -
Fixed a DOCX table cell spanning several grid columns (
w:gridSpan) or rows (w:vMerge) being returned once per covered column and again for every covered row, so a cell merged across 4 columns and 3 rows came back 12 times inresult.tables[].cells,result.tables[].markdown, andresult.contentalike — a 39 KB document could extract to 232 KB. A merged/spanned cell's text is now written once, at its origin, with the columns and rows it covers left blank. This also fixes a DOCX header or footer table with a merged cell shifting every following cell one column to the left (#1549). -
Fixed PDF render diagnostics matching a captured engine warning against a hardcoded message substring to decide whether it meant a glyph actually failed to paint. The message it was built to exclude no longer reaches this capture at all (it moved to TRACE under #1547), so the match could only ever misfire: a future warning whose text happened to share that substring would have been silently dropped instead of surfacing as a
ProcessingWarning. Every captured warning is now reported (#1548). -
Fixed a PDF page that places a statistics table beside a prose column being emitted in full-width Y order, which spliced the prose apart mid-sentence (
more likely to be aged 35Female 51.5 ...) and welded the table's two label/value panels together on every row. The table region is now emitted whole, in row order, ahead of the prose column, and a repeated panel is emitted panel by panel (#1545). -
Fixed PDF text coming back scrambled when a short
Tjrun sat between twoTJarrays: the run was emitted at an earlier run's stale position and sorted into the wrong place, sowithin a period ... after conclusionextracted aswincthin a period ... after co lusion. Every text-showing boundary operator closed the pending run exceptTJ(#1544). -
Fixed every image in a DOCX reporting
page_number1 regardless of the page it sits on. The page was resolved by searching rendered Markdown for a per-image placeholder that is never written -- every drawing renders to the same link target -- so the lookup always missed. Page numbers now come from the parsed element order (#1546). -
Fixed an author's hyphen being deleted when it fell at a line break, so
price-+determiningjoined aspricedetermining. A hyphen written mid-line elsewhere in the same document is now treated as evidence that the compound is real and its hyphen is kept. Compounds that appear only broken, with no such occurrence anywhere in the document, are still joined without the hyphen (#1543). -
Fixed OCR backends registered through
register_ocr_backendbeing rejected before extraction started: configuration validation checked the backend name against the built-in list only, which made every custom plugin OCR backend unusable once validation was wired intoextractandextract_batch. -
Fixed the native C FFI library shipping without eleven features the crate advertises, so the Java, Go, C#, Swift, Zig, and C bindings had no summarization, translation, analysis, HEIC, captioning, ML redaction, or static-embedding support. The desktop dependency hand-maintained a feature list that had drifted from
full; a regression test now fails on any future omission. -
Fixed HTML pages fetched over HTTP(S) losing every format-specific metadata field: results were reported as
text/htmlwhilemetadata.formatstayed empty, because the extraction ran over the crawler's pre-rendered Markdown and never reached the HTML extractor. Title, headings, Open Graph, Twitter card, links, and structured data are now recovered from the page HTML. -
Fixed
pdf_options.hierarchy.enabledsilently producing no hierarchy: headings were detected and then discarded unless the caller also set the unrelatedpages.extract_pages. Requesting the heading hierarchy now enables the per-page tracking it requires. -
Fixed the bundled Tesseract build failing to configure on Windows when the MSVC developer environment is not present, which broke building Xberg from source with the default OCR features.
-
Fixed URL extraction reporting internally converted HTML pages as
text/markdown; results now retain a validated, canonical source MIME type. -
Fixed
clear_post_processorsstopping at the first failed shutdown hook and permanently removing enabled built-ins; it now attempts every shutdown, returns the first error, and restores built-ins before the next post-processed extraction while custom processors remain cleared. -
Fixed VLM concurrency limits increasing concurrent local OCR work and raster memory use (#1465).
-
Fixed structured extraction forcing every caller schema to JSON Schema Draft 2020-12; validation now honors the schema's declared draft while keeping external reference resolution offline (#1539).
-
Fixed hybrid PDF OCR dropping surrounding prose when a table-bearing bare-text page was restructured alongside geometry-backed pages.
-
Fixed automatic PDF OCR fallback reporting an empty success when OCR failed and no native text remained; recoverable failures still return available native text with a warning.
-
Fixed degraded VLM fallback output replacing denser OCR text, while abstaining from the density comparison for short text and non-space-delimited CJK or kana content.
-
Fixed Windows source and Ruby package builds failing on stable Rust while validating the identity of staged Tesseract source directories.
-
Fixed GCC 12+ WordPerfect builds by adding the standard header that declares
size_tbefore compiling the pinned libwpd source. -
Fixed Ruby source-package installation by aligning the Gemfile and lockfile with the gemspec's supported
rb_sysrange. -
Fixed generated Ruby development commands so Bundler and its tools use the active Ruby interpreter, avoiding native-extension ABI conflicts on systems with multiple Ruby versions.
-
Fixed generated Python optional constructor arguments so Pyrefly receives precise keyword types without unused helper declarations.
-
Fixed generated Dart tests for nested tagged unions, nullable payloads, and Flutter Rust Bridge tuple accessors; added e2e analyzer coverage and refreshed the Dart lock file to the generated Flutter Rust Bridge version.
-
Fixed compressed image inputs with oversized declared dimensions exhausting memory during OCR, layout and QR detection, image classification, re-encoding, HEIF conversion, or structured-image rasterization; decoded allocations now obey
security_limits.max_content_sizeand are rejected from the image header before pixel decode. -
Fixed PDF OCR fallback being suppressed for image-only pages when dot leaders or other non-textual native content pushed the document below the alphanumeric-ratio threshold.
-
Fixed process-global native PDF font-cache collisions that made glyph spacing, geometry, and batch output depend on document order and concurrency when fonts used indirect width tables.
-
Fixed Markdown OCR metadata so word counts and confidence statistics describe only text retained after dictionary filtering; fully filtered output now reports zero words and omits confidence quantiles.
-
Fixed repeated bold PDF presenter labels and same-row legend keys being promoted to headings, which could invert document hierarchy and fragment retrieval chunks.
-
Fixed PDF OCR so fragmented, low-confidence, and dictionary-suspect non-empty text is retained with a processing warning by default instead of silently emptying pages. Set
ocr.quality_thresholds.discard_suspected_ocr_noise = true(or the equivalent pipeline quality threshold) to opt into the previous destructive filtering behavior. -
Fixed runtime crashes in system-linked Tesseract OCR builds by linking the required native exception-safety shim.
-
Fixed
xberg batchso mixed-success runs emit every successful document and every attributed per-input error before returning a nonzero status; JSON and TOON timing slots remain aligned with inputs, and TOON now uses the documented batch envelope. -
Fixed
xberg extract --ocr falseso it authoritatively disables implicit OCR fallback, overrides conflicting loaded OCR routing, and rejects contradictory OCR flags. -
Fixed Tesseract preprocessing so deskew, denoise, contrast enhancement, and Otsu, adaptive, and Sauvola binarization settings transform the OCR raster on native and WebAssembly backends;
none(withoffas an alias) preserves unthresholded grayscale when deskew is disabled, sparse receipt-image fallback and faint colored text no longer lose content to global thresholding, dark labels over bright map fills still receive Otsu preprocessing without isolated or clustered dark artifacts triggering it, and WebAssembly Tesseract now rejects images exceeding 4096 × 4096 pixels before decoding. -
Fixed OCR measurement tooling so line-filter comparisons score the intended ground-truth lines and report filtering regressions accurately.
-
Fixed the OpenAPI document's dangling Djot attribute reference so schema validators and client generators can resolve every advertised component (#1505).
-
XML and JSON content with unsupported specialized extensions now routes through the supported generic extractor instead of failing MIME validation (#1507).
-
File extraction now falls back to bounded content sniffing when a path has an unknown or missing extension (#1506).
-
Explicit
application/octet-streamhints now trigger configured MIME detection instead of being treated as an authoritative document type. -
Fixed documentation-snippet fixtures that named non-existent result fields, which made the generated snippets silently drop the affected presentation block: element
contentis nowtext, tablerowsis nowcells, and the result pathskeywords,structured_data, anddocument_structureare nowextracted_keywords,structured_output, anddocument. -
Fixed EPUB extraction for
text/htmlspine items, named entities, declared non-UTF-8 encodings, navigation documents, SVG fallbacks, nested tables, MathML, headings, images, and malformed HTML (#1486, #1488-#1494). -
EPUB extraction now preserves usable chapters when another spine item fails and reports per-item warnings instead of failing the whole document (#1491).
-
Fixed EPUB metadata, EPUB 2/3 cover selection, DRM detection, and font-obfuscation handling (#1492, #1494).
-
Fixed PDF OCR and rendering for highly compressed scans, CCITT images, CFF fonts, maximum-size font tables, malformed embedded fonts, rotated content, missing glyph warnings, and concurrent Pdfium extraction.
-
Fixed native PDF tracing so corrupt optional content is reported as a recoverable warning, while mandatory cross-reference failures emit a single operation-boundary error without changing the returned error type.
-
Fixed annotation-only PDFs so visible FreeText content is recovered into page-aware document text, including when OCR replaces the page text, without exposing hidden, transparent, cropped, or disabled annotations when annotation extraction is off.
-
Fixed the Swift package manifest so SwiftPM no longer warns about a nonexistent target-relative license file.
-
Fixed scanned PDF extraction so CCITT parameters align with their filter in multi-filter streams, referenced JBIG2 image masks are available to OCR, and stencil-mask polarity renders text as opaque.
-
Fixed PDF reading order for dense two-column layouts, hanging clause numbers, split list markers, and modest font-size changes on one baseline.
-
Fixed PDF heading recovery for repeated bold section titles set at body font size while retaining short bold labels, presenter attributions, and calendar legends as body text (#1513).
-
Fixed PDF table extraction so multi-word cells, rule-less prose regions, OCR-derived tables, and page-local table failures are handled correctly (#688, #1358, #1542).
-
Fixed PDF Markdown and Djot output so native text is retained when structured conversion is incomplete.
-
Fixed PDF configuration so metadata suppression and header/footer settings are honored by every backend; invalid or unsupported PDF and OCR settings now return configuration errors.
-
Fixed OCR-backed PDFs so filtering, confidence thresholds, hierarchy, tables, formulas, lists, bounding boxes, page boundaries, and partial page results are preserved consistently across output formats and OCR backends (#1444).
-
Fixed Tesseract caching, configuration, preprocessing, page segmentation, and font-size extraction.
-
Tesseract Markdown extraction now reports a
ProcessingWarningwhen dictionary filtering removes physical text lines, including the number removed. -
OCR element hierarchy output now honors
build_hierarchyand contains only resolvable parent references. -
Fixed Sceptre and PaddleOCR line grouping, region ordering, per-page resizing, table validation, and font-size reporting.
-
Fixed DOCX extraction for nested tables, VML images, text boxes, comments, fields, headings, hyperlinks, headers, footers, table-of-contents entries, nested lists, and page attribution; element output now preserves explicit page breaks and single-page documents report page metadata consistently (#1452, #1460, #1503).
-
Fixed PPTX extraction for malformed relationships, nested image paths, equations, fallback shapes, comments, metadata, and security limits.
-
Fixed spreadsheet extraction for hyperlinks, formulas, names, comments, hidden state, dates, and OpenDocument metadata.
-
Fixed ODT, ODP, iWork, HWP, DBF, RTF, email, and PST extraction across nested content, metadata, binary data, folder traversal, and repeated text.
-
Fixed Markdown, MDX, RST, HTML, DocBook, JATS, FictionBook, Djot, Org, YAML frontmatter, and Jupyter extraction so supported structure and content are retained.
-
Fixed
result.elementsso headings report their level (metadata.additional["heading_level"]) instead of every##-######heading collapsing into indistinguishableHeadingelements with empty metadata;result.document.nodesalready carried the level correctly (#1504). -
Fixed CSV parsing for stray quotes and archive extraction order.
-
Fixed MIME routing so HTML is detected before the generic XML fallback and supported-format lists reflect the active extractor registry.
-
Fixed post-processing, chunking, enrichment, translation, NER, QR codes, captions, and caching so extracted structure is preserved consistently.
-
Fixed chunking presets so standalone and pipeline APIs apply the documented size and overlap while preserving unrelated chunking settings.
-
Fixed extraction timeout handling so timed-out work is cancelled.
-
Fixed configuration merging so changing one CLI option no longer erases sibling settings.
-
Fixed multipart API extraction to accept
jsonanddoctagsvalues foroutput_format. -
Fixed cache keys to reflect only settings that affect the corresponding extraction or OCR result.
-
Fixed model caching so OCR, embedding, and reranking settings no longer reuse incompatible models.
-
Fixed Node.js native-library loading, Swift iOS resolution, Windows DirectML packaging, and
cargo install xberg-cli(#1456). -
Fixed Docker image builds and reduced the CLI image to runtime dependencies.
-
Fixed API and packaging defects in the Python, PHP, Dart, Go, Java, C#, Kotlin, Elixir, Ruby, Zig, and C packages.
-
Fixed Windows wheel and gem packaging, manylinux compatibility, musl smoke tests, and dynamic Tesseract builds (#1495, #1497).
-
Fixed archive and ZIP validation for small compressed entries and impossible declared sizes (#1496).
-
Fixed batch extraction so configured caches are used and progress callbacks report completed items.
-
Fixed extraction configuration validation so invalid nested values, including OCR quality and scanned-page thresholds, are rejected consistently by every public entry point.
-
Fixed error classification so callers can distinguish all documented extraction failure categories.
-
Built-in path and byte extraction now always reports a recognized
extraction_method; custom extractors retain explicit provenance and otherwise leave it unspecified. -
Fixed owned document classification so detected labels are written back to the returned document.
-
Fixed
ContentFilterConfig.include_watermarksso enabling it retains watermark content. -
Fixed
JsonExtractionConfig.flatten_nested_objectsso disabling it preserves nested objects instead of flattening them. -
Fixed standalone-image and OCR-backed PDF results so preprocessing scale, dimensions, and DPI are retained.
-
Fixed Candle OCR configuration so supported backend options are validated and applied.
-
Fixed PaddleOCR-VL so the task selected when constructing the backend is honored unless a request explicitly overrides it.
-
Fixed keyword extraction so invalid n-gram ranges return an error instead of silently producing empty results.
-
Fixed builds that enable only the
apiormcpfeature. -
Fixed the
excel-wasmfeature so spreadsheet extraction builds for WebAssembly. -
Fixed WebAssembly configuration so unsupported managed credential providers are rejected explicitly.
-
Fixed the Swift package failing to link on Linux.
Package.swiftlinkedlibxberg_ffi.aalongsidelibxberg_swift.a, but the Swift static library already folds the entire compiledxberg-fficrate in, so every Rust core, std, and alloc symbol existed twice and the linker reported hundreds of duplicate symbols. It also never asked for ONNX Runtime, leavingOrtGetApiBaseundefined. -
Fixed the public
clear_ocr_backends()andclear_renderers()leaving their process-global registries permanently empty. Afterclear_ocr_backends()every later extraction failed with "No available OCR backends"; afterclear_renderers()theCustomoutput-format path silently downgraded DOT renders to plain text for the life of the process. Both now re-seed the built-ins non-destructively, keeping user-registered entries. -
Fixed nested lists rendering as flat, blank-line-separated bullets in
pages[N].content: container list markers are never page-tagged, so a page subset dropped them and every item was rewrapped in its own single-item list. Also fixed figure alt text being dropped whenever a caption was present, the VLM OCR probe reporting availability without checking credentials, and the PDF margin filter judging rotated text runs by baseline origin. -
Fixed HEIC-enabled builds requiring a libheif newer than current stable distributions ship. The prebuilt artifacts link libheif dynamically and were built against 1.21 APIs, so the PHP extension failed to load on Debian 13 with
undefined symbol: heif_image_get_plane_readonly2. The floor is now 1.19, with version-gated fallbacks (#1541). -
Fixed PDF text collapsing on itself when a font's
/Widthsarray declares 0 for an ordinary glyph. Extraction now falls back to the embedded font's own advance for such codes, while an explicitTJdisplacement stays authoritative and genuine zero-width combining marks remain overlays. -
Fixed automatic PDF OCR replacing a page's native text with a substantially poorer recognition. OCR output for a page whose native text was independently judged healthy is now rejected when it retains under half that page's alphanumeric characters.
-
Fixed OCR of a single detached page image being attributed to page 1. Local image indices were used as document page numbers, so warnings named the wrong page and the rejected-page filter discarded OCR elements, tables, and formulas belonging to a different page than the one rejected.
-
Fixed XML extraction narrowing element depth to
u8before clamping, so an element nested more than 255 levels deep wrapped to a low heading level in release builds and panicked in debug builds, before the configuredmax_xml_depthlimit ever applied (#1474). -
Fixed PDF XMP metadata losing text fragments split around an entity boundary: named and numeric XML references in XMP scalar and sequence values are preserved instead of the surrounding text being truncated (#1475).
-
Fixed image-level OCR running again over a page-sized PDF XObject on a page whose native text had already been extracted, which duplicated the page's content and paid for a second OCR pass (#1479).
-
Fixed the musl (Alpine) native artifacts failing to load. The published Java, C#, Zig, C, and Elixir artifacts shipped without ONNX Runtime's transitive closure — libprotobuf-lite, the
libabsl_*set, libre2, and libicu. Both musl images now vendor the fulllddclosure and hard-fail the build if anything is unresolved. A host runtime that links libstdc++ itself still needs libstdc++ 15 or newer in the process, because a bundled copy cannot win once the soname is already mapped. -
Fixed a DOCX or PPTX relationship targeting
../media/image1.png— the ordinary OPC shape for an image at the package root — being rejected by the traversal check and dropped, so the image went missing from extraction. Container-relative names now resolve boundary-relative. -
Fixed OCR of rendered PDF pages assuming a 72 DPI raster when pages render at 150 DPI, so DPI normalisation computed a 2.48x upscale, hit the dimension clamp, and reported a resolution hint of 179 for what was really a 372 DPI image. Also fixed image DPI normalisation being skipped entirely in candle-backend and VLM-only builds.
-
Fixed layout detection marking real figure and drawing text as page furniture, which the renderer then discarded, so labels such as
SITE PLANandLEGENDdisappeared from scanned documents. APicturehint now means a figure was detected, not that the text is decoration, and furniture hints only match short text. -
Fixed the Docling-compatible endpoint discarding OpenWebUI's extraction parameters. OpenWebUI sends one form field per key rather than a JSON blob, so settings made in its admin UI produced identical output with or without them (#1462).
-
Fixed CLI flags being silently discarded.
--ocr-backend,--ocr-language,--ocr-auto-rotate, and--ocr-backend-optionswere dropped unless--ocr truewas also passed, so--ocr-scanned-pages --ocr-backend sceptreran Tesseract with no error;--ocr-scanned-pagesalone returned an empty document at exit status 0; and--chunk-sizewas a no-op without--chunk true. -
Fixed legacy
.docextraction emitting every field's instruction — its URL, switches, and screen-tips — verbatim as prose, and the non-breaking hyphen being dropped with the other control characters, fusingtwenty-oneintotwentyone. -
Fixed paragraph grouping only breaking when a line starts a numbered section and never when the previous line was one, so a subsection heading followed by unnumbered lines at the same size and weight was merged into the following prose (#1467). Consecutive numbered headings are likewise no longer welded into a single paragraph (#1386).
-
Fixed the PDF pipeline stripping a list item's printed marker and discarding it, leaving renderers to synthesize a position, so a document whose clauses are cross-referenced by their printed label was renumbered —
B.rendering as1.and(a)as1.. -
Fixed
candle-trocraccepting a whole page and returning invented text. TrOCR is trained on single cropped lines and force-resizes any input, so a multi-page document exited successfully with text appearing nowhere in it. Input taller than a plausible line crop is now rejected. -
Fixed inline
<svg>elements being discarded during HTML extraction even withextract_imagesenabled (#745). -
Fixed an explicitly requested GPU execution provider silently running on CPU.
is_available()reports only compile-time support and ORT's session builder defaults to not erroring on failure, so an explicit CUDA, TensorRT, or CoreML request that failed to load was swallowed. Explicit requests now fail;Autokeeps its silent fallback. -
Fixed DOCX documents with legacy VML picture markup being rejected as
NestingTooDeep, and the inverse hole where content inside drawings, table property helpers, the table grid, and streaming section properties was never measured against the depth cap at all. A flat 600-row table of real depth 8 previously leaked over a thousand levels and was rejected outright (#1395). -
Fixed
XBERG_LLM_API_KEYandXBERG_LLM_BASE_URLfabricating a structured-extraction config with an empty model and schema, so any deployment that merely had an LLM key in its environment ran the post-processor on every document and failed every one (#1421). -
Fixed two PDF paths aborting or failing the whole request: a
/ModDatewhose raw bytes decode to a replacement character sliced astroff a char boundary and panicked, which across the Go FFI boundary aborts the process before anycatch_unwindframe is consulted; and a rasterizer panic on a page with damaged content streams unwound through the async boundary and lost every other page's text (#1422, #1408). -
Fixed keyword extraction panicking on a language hint whose first character is multi-byte.
-
Fixed legacy
.pptslide numbering and image extraction. Slide numbers were the ordinal of a text block in a joined string, so a trailing paragraph mark cut one slide into several; they now come from the slide containers in persist order. The OLE/Picturesstream was never opened, so.pptextraction never produced an image (#1418, #1417). -
Fixed PPTX slides without a title losing their page number (#1413).
-
Fixed URL extraction reporting no crawled URLs, because the result field is no longer populated upstream. The URLs are now derived from the crawled pages, deduped in first-seen order.
-
Fixed PDF page-number stripping deleting real table data. The decision was made from one paragraph's text, so any short numeric cell matched; it now requires a margin band, a stable horizontal slot across pages, and a progressive sequence to agree (#1411).
-
Fixed PDF paragraph breaks never being detected on a normally-set page, so a whole memo — date, salutation, body, sign-off — came back as one line. The vertical advance is now compared against the body leading, which is scale-free.
-
Fixed detected PDF tables being injected on top of native text that already contained them, so the same content was rendered twice.
-
Fixed non-HTML raw blocks being written verbatim into styled HTML output. ODP speaker notes and master-page text, Org source, script and style bodies, and Djot raw blocks all reached the page unescaped, so any
<in them corrupted the document structure. -
Fixed the PyPI
xberg-cliwheels shipping without their native libraries. The build hook force-included siblings with a macOS-only glob, so every Linux shared object staged beside the binary was dropped, and the musl wheel shipped only the launcher script. An incomplete platform payload now fails the build instead of publishing a wheel that installs and cannot run. -
Fixed OCR'd PDF pages reporting bounding boxes in raster pixels while digital pages report PDF points, with nothing in the response distinguishing the two spaces. Node, hierarchy block, chunk page span, and table bounding boxes are now converted to page points with a bottom-left origin (#1423).
-
Fixed OCR on pages carrying a
/Rotateentry. Backends now declare how they cope with a rotated raster, so a backend that requires an upright page is handed one with its geometry mapped back, and PaddleOCR receives the page rotation as a sort key. Auto-rotation composes with the page hint instead of double-correcting it. -
Fixed PDF text and tables on rotated pages. Rotated-text repair reconstructs the reading frame but only when rotated spans are at least 20% of a page's characters, so a single rotated caption no longer costs the upright majority of the page its whitespace structure, and heuristic table reconstruction clusters cells on the table's own axes rather than raw page space (#1358).
-
Fixed the OpenAPI document omitting types that client generators need: second-order nested component schemas are now registered, along with the PDF, office, and transcription schema groups and the
415and429responses the extraction endpoints can return (#1424). -
Fixed
code_intelligencebeing hardcoded toNone, so the documented metrics, imports and exports, comments, docstrings, symbols, and diagnostics never reached callers (#259). -
Fixed Whisper timestamp tokens leaking into transcripts as literal text. They are not marked special in the tokenizer vocabulary, so they survived decoding; they are now paired into segments, emitting one paragraph per segment with start and end times.
-
Fixed
cargo add xberg --features fullfailing to link on Windows MSVC, where a transitive build script forces/MTwhile Rust defaults to/MD, killing the build withLNK2038(#1389). -
Fixed
show_download_progresshaving no readers anywhere on the embedding, sparse-embedding, reranker, and late-interaction model configs, so the documented option did nothing. -
Fixed
split_and_extractrebuilding each segment from a handful of fields, dropping keywords, entities, summaries, chunks, warnings, and the rest of the enrichment that extraction produced, and an off-by-one in the chunk image-index remap that pointed chunks at the wrong image. -
Fixed
target_dpi,max_image_dimension,auto_adjust_dpi,min_dpi, andmax_dpihaving no readers: every preprocessing config was built with defaults, so these settings were dropped (#209). -
Fixed declared telemetry that never emitted. The cache-hit, cache-miss, and batch instruments were declared but never recorded, and the pipeline and batch operations, five of the eight pipeline stage spans, and the extractor-priority and batch attributes were likewise never recorded, so filtering on them returned nothing (#332, #282).
-
Fixed an injected cache backend never being consulted and
ProgressSink::emithaving no caller on single extraction;extract_batchwas already correct. A bytes-input cache hit now short-circuits extraction and coarse start, complete, error, and cache-hit events are emitted. -
Fixed renderer output completeness: JSON silently dropped page breaks, footnote references and definitions, citations, slides, definition terms, admonitions, raw blocks, and metadata blocks through a catch-all arm; styled HTML opened a section for each slide that was never closed and never rendered the slide title; and formulas rendered as preformatted code, which KaTeX and MathJax cannot pick up, and are now delimited display math.
-
Fixed footnote definitions never appearing in JSON output, and a definition present in the document but never referenced being dropped from rendered output entirely (#68).
-
Fixed plugin-produced documents losing content at the bridge. The conversion into the internal document dropped
uris,children,annotations,processing_warnings,llm_usage,pages, andocr_elements; native renderers reached through the public entry point emitted an empty shell; andpre_rendered_contentwas ignored for HTML and JSON output. -
Fixed CRLF documents collapsing into a single paragraph. Ten call sites split paragraphs on a bare double newline without normalising line endings first, affecting email and PST bodies, OCR backend output, plain text, and Djot conversion (#227).
-
Fixed MIME aliases that were advertised as supported and then failed as
UnsupportedFormat, because the registry looks up by exact string with no alias resolution.application/wordperfect,application/x-quarto, and four audio and video transcription aliases now route to the same extractor as their canonical type. -
Fixed three internal OCR plumbing keys being copied into user-visible document metadata.
-
Bounded DOCX image and iWork archive member reads by the member's declared uncompressed size instead of trusting that declaration. A crafted document could forge a small declared size in the ZIP central directory while carrying a deflate stream that inflated to multiple gigabytes, exhausting memory during DOCX image extraction (
images.extract_images) or.pages/.numbers/.keyextraction. Reported by Syed Anas Mohiuddin (GHSA-85w9-wqcq-x48r). -
Pinned downloaded Tesseract, Leptonica, and English tessdata inputs to immutable revisions with verified sizes and SHA-256 digests, race-safe content-addressed caches, private build directories, and bounded fail-closed archive extraction.
-
Structured extraction now resolves caller-provided JSON Schemas strictly offline and rejects external HTTP and file references without performing I/O.
-
REST and MCP requests can no longer override LLM credentials, provider registrations, or other server-controlled settings.
-
Hardened ZIP accounting against overflow, impossible sizes, and compression-ratio bypasses.
-
Hardened DOCX, PPTX, and EPUB relationship resolution against container traversal, malformed UTF-8, NUL bytes, drive-letter paths, UNC paths, and symlink escapes.
-
Added bounded EPUB traversal and retained-content accounting to prevent resource-limit bypasses.
-
Cache namespaces are validated before directories are created.
-
Redaction now reports only content that was actually removed, never exposes pre-redaction element text, and rejects invalid strategies instead of silently falling back to masking.
-
Hardened the native PDF engine against crafted documents that abort or hang the host process. A self-referencing
/Names /EmbeddedFilestree and deeply nested array or dictionary brackets each recursed until the stack overflowed, which is an abort nocatch_unwindcan contain; a negative/Welement in an xref stream, a reversedbfrange, a non-hexToUnicodedestination, an all-NaN font-size set, and unchecked/Widthx/Height,/N, and/VerticesPerRowproducts each panicked or allocated without bound; anddecode_stream_with_params, the entry point every production call site uses, applied no ratio or size guard at all. All were reachable fromextract_bytesunder default configuration. -
Bounded every ZIP, TAR, and 7z member read against
SecurityLimitsrather than against the size the archive declares for itself, since a declared uncompressed size is not a bound and the aggregate check previously ran only after the member was fully resident. Covers generic archives, ODT, ODP, EPUB, HWPX, PPTX, XLSX, and OOXML embedded objects, and adds the compression-ratio and aggregate-size validation that PPTX, XLSX, and DOCX were missing. A nested ZIP no longer overflows the stack. -
Clamped or rejected document-declared counts that reached an allocation or a slice unchecked: HWP table row and column counts, HTML and EPUB
colspan/rowspan, DOCXw:ilvl,w:gridSpan, andw:outlineLvl, PPTXa:pPr lvl, RST simple-table column ranges, JATSdate-type, EPUB link-label offsets, PPTX relationship targets, and the hOCR parser's and annotated-text renderer's byte-offset slices. Each was an out-of-bounds or char-boundary panic, or an allocation abort, on ordinary untrusted input. -
security_limits.max_files_in_archiveis now enforced by every OOXML container. XLSX never checked it, DOCX enforced a hardcoded 10,000-entry cap instead of the configured one, PPTX had no entry check at all, and embedded-object extraction walked embeddings uncapped (#1449). -
EPUB packaging XML now counts real OPF nesting depth against the configured limit and accepts legacy DTD declarations without resolving external or amplified entities, so a crafted package can neither bypass the depth budget nor pull in outside content (#1477, #1478).
-
Native PDF tracing no longer carries document content. Decoded page text was emitted verbatim at TRACE, embedded font names appeared in trace events and in the glyph-drop
ProcessingWarningmessage, and parser, xref, and recovery failures were logged by formatting the underlying error string. Failure paths now emit a structurederror_codewith an optional byteerror_offset, and font names are redacted in the warning text. -
Bounded the native PDF reader's internal caches so a malformed or hostile document cannot grow them without limit: the object-stream cache evicts to a byte budget and rejects oversized entries, font identity hashing stops at a byte budget and a reference-depth cap (both recorded in the hash so distinct fonts stay distinct), and the xref recovery-marker set is capped.
- OpenWebUI-compatible endpoints (
PUT /processandPOST /v1/convert/file) now honor extraction configuration. They previously cloned the server default, forced Markdown output, and ignored all inbound parameters, so configuration passed through OpenWebUI had no effect. They now use the server's configured defaults as the base and merge a per-request config — a multipartconfig/parametersfield, or theX-Configheader — matching the/extractendpoint, keeping Markdown as the default only when neither the server config nor the request selects a format. - Image captioning is now included in the official Docker images (
--features all), and the server emits aProcessingWarningwhen acaptioningconfig is supplied but the feature is compiled out, instead of silently doing nothing (#1382). - Release builds no longer check out the
test_documentsbenchmark submodule, so a benchmark-only submodule update can no longer fail every publish build and ship a release with no assets (#1380).
- Embedded-image captioning now runs with bounded concurrency (mirroring the image-OCR path) instead of one VLM request at a time, reducing wall-clock time on image-heavy documents (#1378).
- OCR-backed PDF extraction now keeps consecutive Tesseract paragraphs grouped within their shared hOCR text area instead of splitting them, including pages replaced by mixed native/OCR extraction. This is paragraph/block grouping only; font-clustering headings and list-marker detection for OCR-backed pages are tracked separately (see Unreleased).
- PDF table reconstruction now rejects sparse, short-wide contact blocks that were previously misclassified as tables.
- Standalone-image Tesseract OCR now defaults to sparse-text segmentation, while cropped layout
regions use single-block segmentation and explicit user settings remain unchanged. Vertical
language packs such as Japanese (
jpn_vert) use vertical-block segmentation. - Standalone image extraction now reports successful OCR through
metadata.ocr_usedand the OCR extraction method, including layout-aware OCR results. - Tesseract now applies its default image preprocessing only to clean, near-white document pages; shadowed receipts and photographic images keep their source pixels, avoiding quality loss from destructive DPI upscaling, background normalization, sharpening, and grayscale conversion.
- Sparse, low-confidence standalone Tesseract results now retry the previous automatic page segmentation with explicit preprocessing and use it only when word confidence is consistently strong, recovering difficult receipts and scene text without replacing reliable sparse output.
- CSV and TSV plaintext now use the canonical table renderer instead of lossy
Row Nand header-value prose. - Extracted EML and MSG attachment text is now included in the parent document while the structured attachment children remain available.
- DOCX extraction now emits a tab character for an in-run
<w:tab/>instead of dropping it, so tab-separated fields — most visibly Word table-of-contents rows — no longer weld adjacent words together (Alpha<tab>Betawas extracted asAlphaBeta). Tab-stop definitions remain invisible. (#1377) - The Swift package builds and publishes again. The cross-compiled desktop
xberg-ffidependency no longer pulls in HEIC (libheif-sys, which has no cross-compile support) or the Candle OCR backends, which had broken Swift package publishing in 1.0.12. - The NuGet runtime packages for macOS and Linux (
osx-x64,osx-arm64,linux-x64,linux-arm64) now publish at the current version instead of being stuck at an older one; previously only the Windows runtime package was updated. (#1375) - The public in-browser (WASM) demo now attributes its file-size limit to the browser sandbox and points to the CLI and API for large or multi-page documents, instead of implying the document itself is at fault. (#1376)
- The
xberg mcpextractandextract_batchtools no longer emit structured output that fails their own declared output schema. The schema requirederrorsand thecrawl_*fields, but a normal extraction omits them when empty, so MCP clients (e.g. Claude Code) rejected the result. Those fields are now optional in the schema, matching the serialized output. (#1372) - The
install.shscript no longer creates a self-referentialxbergsymlink that shadowed the installed binary, and it now selects the glibc (-gnu) build on standard Linux distributions instead of always downloading the musl build — which failed to run on glibc systems such as Ubuntu. musl systems (e.g. Alpine) still get the musl build. (#1371) - CSV header inference no longer misclassifies all-text tables as headerless. A first row such as
Name,Cityis now treated as the header (the dominant CSV convention) instead of rendering a broken blank header row with the real header pushed down into the data. A numeric-looking first row is still treated as data. (#1369)
- The extraction HTTP server now bounds in-flight request concurrency so a burst of large uploads
can no longer exhaust memory and OOM-kill the process in memory-limited containers. The limit
defaults to
2 × CPU countclamped to[4, 32]; override it withXBERG_MAX_CONCURRENT_REQUESTS(set0to disable). (#1368) - PaddleOCR output now keeps consecutive visual text lines in the same Markdown paragraph instead of turning every detected line into a separate paragraph.
- PaddleOCR and Tesseract automatic image rotation now use the document-orientation model's RGB input and existing probability output correctly, and recover sparse edge-aligned text that the model's standard center crop omitted.
cargo install xberg-clinow succeeds on a stock Windows toolchain. HEIC/HEIF decoding links nativelibheif, which has no default build path on Windows, so it is no longer part of the CLI's default features and the install no longer fails buildinglibheif-sys. Enable HEIC with--features heic; the prebuilt release binaries, Dockerallimage, and Homebrew bottle continue to ship it. (#1361)- The
cargo binstall xberg-clistatic musl builds now compile. The #1355 image-fallback OCR helpers were gated on theocrfeature but are reachable under theocr-pipeline-onlybinstallprofile, which failed to build both musl targets in the 1.0.9 release. brew install xberg-io/tap/xberginstalls a working binary again instead of an empty bottle; the 1.0.9 bottle rebuild had been skipped when the CLI asset upload cascaded from the failed binstall build. (#1356)- The hosted demo page (docs.xberg.io/demo.html) no longer 404s its toolbar and file-picker icons. (#1360)
- Dart native-library loading now propagates download, filesystem, and checksum failures instead of silently falling back to an unverified default library resolution path.
- PaddleOCR concurrent cold starts now run off async worker threads and share one engine initialization per model and accelerator, with distinct cache entries for different GPU device IDs.
- Benchmark text F1 now segments CJK around embedded Latin and numeric text while ignoring OCR line wrapping, preventing mixed-script output formatting from distorting quality comparisons.
cargo binstall xberg-clinow installs a self-contained, fully static musl CLI binary with no ONNX/Tesseract/libheif runtime dependencies. Thex86_64-unknown-linux-muslbuild additionally bundles the pure-Rust Candle VLM OCR backends (TrOCR and PaddleOCR-VL);aarch64-unknown-linux-muslships extraction-only. ONNX/Tesseract/HEIC OCR remain available via Homebrew and the bundled per-target release tarballs.
- PaddleOCR now exposes the
PaddleOcrEnginename and detailed word-level quadrilaterals; the formerOcrLitename remains available as a deprecated compatibility alias. - Dense XLSX extraction now scans worksheet bounds without cloning every cell before normal range parsing, while oversized sparse sheets materialize their cells only once.
- Layout-enabled image table recognition now shares its decoded RGB raster with the TATR worker, avoiding one full image allocation and pixel-buffer copy per qualifying image.
- Multi-stage PDF OCR now shares rendered page rasters across pipeline tasks instead of copying each pixel buffer, reducing peak memory by roughly one RGB raster per concurrent page.
- Batch DOCX extraction reuses one owned input buffer and avoids rebuilding discarded document structure, reducing memory copies and structure-processing overhead for large files.
- Canonical PaddleOCR benchmark presets no longer force optional whole-image auto-rotation, avoiding confident but incorrect 180-degree rotations that suppressed scene-text quality.
- PaddleOCR now preserves native resolution for 1024-pixel images by default, improving scene-text accuracy while retaining explicit detector-size overrides.
- Layout-enabled image OCR now reuses successful single-frame whole-image text when structured assembly is unavailable, avoiding repeated region OCR and redundant Tesseract table analysis.
- PaddleOCR-only CLI builds no longer compile PDF Markdown layout reuse code when layout detection is disabled.
- The prebuilt macOS CLI tarballs (
aarch64-apple-darwin,x86_64-apple-darwin) now bundle the full libheif dynamic-library closure beside thexbergbinary and rewrite its load commands to@loader_path, so the binary no longer fails with alibheif.1.dylibnot-loaded error on machines that lack Homebrew's libheif at the baked-in path (#1357). - PaddleOCR layout and table consumers now use projected CTC word boxes while preserving line-level semantic text and caller-requested element granularity, avoiding mixed-level duplicate table text.
- PaddleOCR detection now honors its configured DB threshold and matches upstream dilation, perspective-crop, and visual-line ordering behavior, improving small, skewed, and jittered text.
- Apple Keynote packages containing only slide archives now route to the Keynote extractor, and Numbers extraction reconstructs tables instead of emitting raw protobuf fragments.
- AsciiDoc, NXML/JATS, and WebVTT files now route through their registered text or JATS extractors instead of being reported as unsupported.
- Standalone
excelandexcel-wasmfeature builds now include the XML parsing and table-capacity support required by XLSX extraction. - Org-mode extraction now distinguishes separator-defined table headers from headerless tables, preserving every data row in rendered Markdown.
- EPUB extraction now resolves
epub:switchbranches per output renderer, preserving supported XHTML and MathML cases while retaining readable plain-text fallbacks. - Typst extraction now emits marker-free headings and distinguishes explicit table headers from bare table rows, preserving correct Markdown structure.
- MSG extraction now reads the canonical binary
PidTagHtmlstream with the Internet codepage, preserving HTML-only message bodies alongside attachments. - TATR table reconstruction now assigns each selected OCR word exactly once, using the nearest cell when predicted cells do not overlap, preventing both duplicated and silently dropped text.
- Layout-enabled image extraction now recognizes TATR table structure from cached OCR elements while preserving non-table line structure and requiring complete OCR token retention before accepting the reconstructed layout.
- Layout-enabled OCR now preserves detected image headings without losing or reordering fallback text, and regroups adjacent PDF OCR lines without collapsing distant paragraphs or separate layout regions.
- Rotated PDF OCR now avoids reusing display-coordinate Markdown layout rasters and reruns layout
on inverse-
/Rotate-normalized images, keeping OCR upright without desynchronizing detections. - Benchmark text F1 treats OCR-inserted line breaks within CJK text as layout whitespace, preventing semantically identical Chinese, Japanese, and Korean output from scoring zero.
- Pipeline quality benchmarks allow forced OCR inference enough time to finish instead of recording slow but valid OCR documents as zero-quality timeout failures.
- Benchmark fixture validation now accepts descriptor filenames without an explicit parent path.
- Pipeline benchmarks now preserve exact ordered cohort fixture paths, use explicit PP-OCR model identities and fixture OCR languages, and score structural image ground truth.
- PaddleOCR now reports processed image dimensions and applied orientation corrections, keeping OCR geometry aligned with optional layout detection on rotated documents.
- PaddleOCR now selects the Japanese model for vertical Japanese and prefers Korean or Japanese recognition for mixed Latin-script requests those models can cover.
- PP-OCRv6 requests containing Korean now use PaddleOCR's script-specific Korean recognizer, recovering Hangul text that the unified recognition model omitted.
- PaddleOCR now preserves the right-to-left column order and contiguous text of traditional vertical Chinese and Japanese documents.
- Image OCR now preserves blank-line paragraph boundaries instead of flattening every recognized text block into one paragraph.
- Tesseract vertical CJK OCR now removes artificial spaces between adjacent script characters while preserving Latin-word and paragraph whitespace.
- Jupyter notebook paths retain
application/x-ipynb+jsonrouting when generic JSON content detection runs, and extracted notebook content no longer exposes diagnostic cell/output markers; cell identity, execution, tag, output-type, and MIME details remain available as structured metadata.
- Candle VLM OCR backends now ship in the published packages. The pure-Rust Candle OCR
backends — TrOCR, PaddleOCR-VL, GLM-OCR, and DeepSeek-OCR — are compiled into the published
packages by default (Python, Node, Go, Java, C#, Ruby, PHP, Elixir, Kotlin/JVM, Zig, and the
CLI / Docker image) on Linux, macOS, and Windows. Select one with
ocr.backend = "candle-glm-ocr"(orcandle-trocr/candle-paddleocr-vl/candle-deepseek-ocr); model weights download from Hugging Face on first use. Previously these backends were excluded from thefullfeature and reachable only via a custom source build. Not available on WebAssembly, Android, iOS, Dart, or Swift.
- #1355 —
force_ocrno longer emits a silently blank page when the PDF rasterizer cannot draw an image XObject. When aforce_ocrpage renders blank but carries image XObjects, OCR is retried directly on the embedded image bytes (decoded pixels, or the raw JPEG/JP2 stream) and a processing warning is recorded, so the page content is recovered instead of dropped without notice. - Swift artifact-bundle cross-compile: the cross-compiled Swift binary bundle builds again —
the HEIC path (which shells out to
pkg-configand cannot cross-compile) is dropped from the Swift / Intel-macOS cross-build feature set (full-no-heic), restoring thex86_64-apple-darwinand Linux Swift builds. The native C FFI distribution keeps HEIC. - XLSX extraction on Windows: the
excelfeature is enabled in the Windows feature set, so.xlsxfiles extract on Windows instead of returningUnsupportedFormatfor a format the registry advertises as supported. - Benchmark CI validates ground truth for every format family plus the exact 101-cell workflow matrix and harness contracts before expensive jobs, and the local benchmark task now delegates to the same run wrapper.
- Benchmark quality rankings and Pareto SF1 multiply successful-extraction medians by accountable coverage exactly once, so partial framework failures cannot retain a perfect rank while harness and setup failures remain excluded.
- Benchmark runs abort instead of dropping task errors, verify exact eligible-document cardinality before writing artifacts, reject contradictory failure states and unknown pipeline names, and report extension success rates with the same accountable-failure semantics as the aggregate.
- Benchmark CI invalidates its prebuilt harness cache for harness build scripts, workspace and toolchain configuration, compiler/codegen environment, and every transitive workspace crate, preventing stale binaries. Release tokens now default to the current repository installation.
- Present best-effort benchmark artifacts receive the same provenance, supported-format cardinality, failure-accounting, and aggregate integrity validation as required artifacts; only absence and framework-accountable extraction failures remain optional. Consolidated provenance, metadata, failure summaries, and rankings are cross-checked against validated groups and rows, with ranking optionality derived from the active cohort rather than a global framework union.
- Subprocess benchmark results record the framework's declared supported extensions, preserving the capability context needed to interpret historical multi-format aggregates.
- Benchmark quality guardrails fail on missing contracted documents or pipeline results instead of reporting a vacuous pass, and reject unknown pipelines, empty predicates, and invalid thresholds before execution.
- Unstructured benchmark cells advertise only their supported plaintext output, and pipeline benchmarks reject unknown sort metrics instead of silently falling back to SF1.
- Benchmark fixtures reject document and ground-truth paths that escape the repository or standalone fixture trust boundary, including symlink escapes, and derive repository boundaries from runtime fixture locations so cached binaries remain portable across CI runners. Artifact provenance hashing uses the same validated path resolution.
- Benchmark CI records declared per-framework format support, validates partial-run thresholds before execution, evaluates them independently per framework, and excludes harness/setup errors from framework success rates while retaining strict extraction coverage for every xberg pipeline.
- Benchmark comparisons include formats with text-only ground truth, report structural scores as unavailable instead of zero when Markdown ground truth is absent, and identify guardrails by file type so same-named fixtures cannot be matched across formats. Guardrails are rebased against the active corpus, removing retired PDF contracts and covering every actionable current result.
- Image layout extraction reuses safely positioned whole-image OCR elements, falls back when region-based OCR drops substantial text or quality, and preserves warnings without redundant OCR retries.
- EPUB extraction removes duplicated serialized MathML and embedded-media fallback content, and avoids emitting a cover image twice when the spine already references it.
- Email extraction preserves sender display names alongside addresses, and asynchronous attachment and nested-message extraction reuses the initial parse instead of parsing messages twice.
- FB2 and DocBook files with generic XML signatures retain their extension-specific MIME types, so they route through the semantic FictionBook and DocBook extractors instead of the generic XML fallback.
- Nested objects and arrays in JSON documents render as structured Markdown headings and lists instead of opaque compact-JSON strings, preserving readable nested keys and values.
- libwpd (Windows/MSVC): link the vcpkg-provided static zlib so librevenge's
inflate*symbols resolve at the final link. Windows binding builds previously failed withundefined symbol: inflatebecause the MSVC path emitted no usable zlib link directive. - #1344 follow-up: Automatic PDF layout inference retries once on CPU only for runtime inference
failures, keeps explicitly selected non-Auto providers and recognized
XBERG_ORT_EPvalues authoritative, ignores blank or unrecognized environment values, and propagates the effective or recovered CPU provider to downstream TATR and OCR table reconstruction. - Side-by-side PDF TATR tables match source words that narrowly cross a detected outer edge to the outermost cell without changing the inference crop or center seam, preserving financial-table row prefixes that previously fell just outside the recognized cell bounds.
- Upgrade sibling dependencies:
crawlberg1.0.11 → 1.1.0,html-to-markdown-rs3.9 → 3.10,liter-llm1.11 → 1.12.liter-llm1.12 makestracingan always-on dependency and removed itstracingCargo feature, so it is dropped from the dependency declaration (no behavior change — liter-llm spans are always emitted now). - The
otelfeature now forwards tocrawlbergandliter-llm(weak,crawlberg?/otel/liter-llm?/otel), so enablingxberg/otelcompiles those siblings' direct OpenTelemetry integration (crawlberg's semconv/propagation, liter-llm'sgen_ai.*metrics); their spans and metrics are exported by the host's provider (e.g. xberg-enterprise).html-to-markdown-rsandtree-sitter-language-packare puretracingemitters with nootelfeature — their spans reach the collector through the consumer'stracing-opentelemetrylayer, so nothing is forwarded to them.
xberg-libwpdWindows build: the WordPerfect extractor now compiles and links onx86_64-pc-windows-msvc, unblocking the full-feature Windows binary of downstream consumers. Two first-ship gaps in the vendored C++ build are fixed: (1) zlib (needed by librevenge'sRVNGZipStream) is now built from source vialibz-syson Windows too — as it already was on Linux/macOS — instead of relying on a vcpkg-installed zlib that CI did not reliably provide (fatal error C1083: Cannot open include file: 'zlib.h'); and (2) a narrowingstd::make_shared<WP6SubDocument>(…, m_streamData.size())call inWP6GeneralTextPacket.cpp— a 64-bitsize()into a 32-bitconst unsignedparam — is patched to cast(unsigned)(matching every sibling subdocument site), which the newest MSVC toolchain (14.5x) otherwise rejects as a hard error. The vcpkg zlib probing inbuild.rsis removed.- #1345: Sparse native two-column PDFs preserve column-block reading order instead of interleaving their four text lines row-by-row across the gutter.
- #1346: PaddleOCR emits a
ProcessingWarningwhen requested languages are not covered by the single selected recognition model (previously their text was silently dropped), and OCR metadata now reports the recognition model actually used instead of joining every requested language. - #1344: Layout inference no longer silently degrades to no-layout output when a hardware
execution provider fails. macOS
autoacceleration resolves RT-DETR to CPU up front (its current export cannot execute under CoreML), so the common path never attempts a failing provider. When an explicit accelerated provider does fail at inference (for example a CoreMLExecuteKernelerror), both the markdown and OCR layout paths retry once on the always-available CPU provider and recover the layout, and either way surface aProcessingWarning(recovered-on-CPU, or lost entirely if CPU also fails) instead of returning byte-identical no-layout output with emptyprocessing_warnings. - #1349: Successful TATR table reconstruction no longer writes source cell content and coordinates to stderr; the debug output is removed.
- #1350: The Markdown hierarchy no longer merges a distant header and footer into one block — paragraph continuation now rejects merges across a large vertical baseline gap and recomputes the merged block's bounding box.
- #1351: The published Node package ships the alef-generated
index.d.ts(clean, consistent types) rather than the rawnapi buildoutput, which emitted references to undefinedJs*types. - #1353: The install script copies nested runtime library directories (for example
lib/libheif) withcp -R, instead of failing withcp: -r not specified; omitting directoryon Linux musl installs.
- Raw
println!/eprintln!/print!/eprint!/dbg!are now denied in production code across the whole workspace (clippyprint_stdout/print_stderr/dbg_macro);tracingis the sole diagnostic surface. The CLI's machine-readable result output to stdout opts back in per call site (#[expect(clippy::print_stdout)]), and the regenerated language bindings route their FFI-bridge diagnostics throughtracinginstead ofeprintln!. - Internal diagnostics that previously wrote to stderr via
eprintln!(per-page OCR gate decisions, GLM-OCR debug tensor stats, the CLI--output-formatdeprecation notice) now emit throughtracingat the appropriate level, so verbosity is controlled withRUST_LOG/--log-levelinstead of ad-hocXBERG_DEBUG_OCR/XBERG_GLM_DEBUGenvironment variables. - Repeated per-page and per-backend warnings from external dependencies (OCR engines, layout models)
are now de-duplicated by
(source, message), so an N-page document surfaces one warning per distinct problem rather than N copies. The paddle-ocr uncovered-language warning is also logged.
- MCP clients can run
extract,extract_batch, andcache_warmas cancellable SEP-2663 tasks when they advertise task support; synchronous clients remain compatible. - MCP
cache_clearandcache_warmreturn typed structured results with cleared-file totals and model availability separated from confirmed cache-hit and download status.
- Dependency bumps:
crawlberg1.0.11,tree-sitter-language-pack1.13.6,base640.23 (xberg-jni).
- #1338: Default
OcrStrategy::Autoextraction OCRs scanned PDFs with no native text layer instead of returning empty content; explicit OCR disablement remains authoritative. - #1341: Synthesized VLM fallback pipelines run for mixed native/OCR PDFs, preserve skipped and failed-stage diagnostics, and retain the last non-empty fallback when every stage scores below threshold.
- #1340: PDF images and generated captions render at bounding-box-aware reading-order positions, remain within the correct layout column, preserve source order, and stay consistent through chunking, translation, and redaction.
- #1343: Archive extraction skips macOS/tooling metadata entries (
__MACOSX/, AppleDouble._*,.DS_Store,Thumbs.db,desktop.ini,__pycache__/,.pyc/.pyo) instead of emitting them astext/plainchildren, and unsniffable extensionless members default toapplication/octet-stream; a single aggregated warning records what was filtered. - Per-file OCR language overrides now also apply to explicit Tesseract pipeline stages, preserving override precedence.
- PDF plain-text extraction repairs detached subscripts, phone suffixes, and final glyphs while preserving RTL, rotated, vertical-writing, and mathematical span order.
- PDF Markdown atomically replaces adjacent native side-by-side table cohorts with validated layout table cohorts, avoiding mixed grids and dropped financial-table structure.
- OCR Markdown applies layout hints to line-local geometry while preserving soft-wrapped body paragraphs and merging multi-line headings, code, pictures, and wrapped list items by hint.
- Tesseract OCR Markdown aligns layout hints and table-cell matching with DPI-normalized and auto-rotated image coordinates, restoring semantic structure on scanned PDFs.
- OCR Markdown recovers missing ordered-list successors only when an existing numeric list item anchors a complete, bounded three-item sequence across pages.
- PDF Markdown preserves strong native headings when a lower-confidence layout Code hint lacks structured code evidence.
- OCR Markdown recovers a title from a guarded first-block logo/title pattern when the layout model emits no semantic heading region.
- PDF Markdown preserves native heading, list, code, and formula semantics while using layout geometry for reading order, grouping, and tables, tolerates minor crop jitter in side-by-side cohorts, merges sparse currency-affix columns without dropping markers, and folds wrapped financial-table lines into logical records; table-dominant pages also discard bbox-confirmed crop spill while retaining surrounding prose and annotations.
- PDF Markdown reconstructs paired wrapped financial tables as semantic three-column grids and repairs consistently merged numeric columns from native PDF table detection.
- #1342: PDF table reconstruction retains short numeric grids when a small number of inferred columns make the principal data row nearly complete instead of fully populated.
- PDF Markdown recognizes repeated large-font heading tiers across sparse multi-page documents while retaining the single-page sparse-document safeguard against display-text false positives.
- PDF benchmark fixtures can pin Tesseract OCR languages. The benchmark harness validates language codes, checks required packs before timed extraction, and preserves the effective OCR backend and cache settings when applying per-file batch overrides.
- Upgraded
rmcpto 3.0.0 and migrated the MCP server to its 3.0 API (schema output, the new cache-scope/result-type/TTL list-result fields, and theGetPromptResponse/ReadResourceResponsehandler enums). The exposed tools, prompts, and resources are unchanged. - OCR now emits
tracinglogs when it materializes a Tesseract language pack at runtime: an info line naming the language, destination, and source before the download, one per candidate URL as it is tried, and one on success. Previously a runtime language-pack download was silent, making a first-use OCR stall on a missing pack hard to diagnose. English is unaffected on builds with thebundle-tessdata-engfeature (embedded, no download). - Dependency bumps:
liter-llm1.11.4,toml1.1.4.
- #1333: A sparse continuation row no longer dilutes the numeric ratio used to classify a grid, so numeric line-item tables with a trailing partial row are kept as tables instead of being flattened to prose.
- #1336: Tesseract no longer creates OCR cache directories when caching is disabled; the cache directory is created lazily, only when a result is written.
- #1337: Light-text-on-dark-background scans are auto-inverted before OCR via mean-luminance
polarity detection, and the previously-dead
invert_colorsconfig is honored as an explicit override (Someforces,Noneauto-detects). - #1338: NER and summarization processors are now compiled into the container and CLI builds — they
were feature-gated out, so
ner/summarizationconfig was silently dropped. UnderOcrStrategy::ScannedPages, a whole-document text failure now OCRs every page instead of discarding the signal. - #1339: VLM OCR forwards
XBERG_LLM_*env credentials toocr.vlm_configwhen a custombase_urlis set, normalizes openai.com model names, routes bare images through the OCR pipeline sovlm_fallbackon_low_qualityfires, and surfaces per-stage OCR failures as processing warnings. - Linux builds without CUDA or TensorRT no longer fail under strict warning settings because of an unused ONNX Runtime execution-provider trait import.
- Per-file OCR language overrides (CLI and benchmark) now reach nested Tesseract configurations.
- PDF plain-text extraction repairs detached text spans so words are no longer split mid-token.
- PDF plain-text extraction retains table assets without rendering native table text twice.
- PDF extraction recovers and stitches label-heavy financial tables without merging independent aligned tables.
- PDF Markdown preserves explicit word boundaries and changelog heading hierarchy.
- OCR Markdown prefers validated semantic layout hints over broad text regions at comparable overlap.
1.0.2 is a packaging release. It completes the 1.0.1 rollout — the PHP/Packagist binding failed to build for 1.0.1 — and adds a first-party coding-agent plugin. No core extraction behavior changed.
- Coding-agent plugin. A first-party xberg plugin for Claude Code, Codex, Cursor, and OpenCode,
with a Hermes variant, ships extraction skills (batch extraction, chunking, OCR, tables, keywords,
format selection) that drive xberg through its MCP/CLI surface. Published as
@xberg-io/opencode-xberg(npm) andxberg-hermes-plugin(PyPI).
- The PHP binding now builds against
ort2.0.0-rc.13. rc.13 moved the CoreML/CUDA/TensorRT execution-provider types behind matching Cargo features; a fresh dependency resolution (as on the PHP build) picked up rc.13 and failed to compile. Those EP features are now enabled unconditionally — a compile-time#[cfg]unlock only, with no SDK dependency or runtime change — so the PHP/Packagist package publishes again.
- Drops the Node
@xberg-io/xberg-win32-arm64-msvcsub-package. It was declared as an optional platform dependency but never built — no xberg binding targets Windows on ARM64 — leaving an unresolvable optional dependency. The target is removed from the package manifest and loader for parity with the other bindings. - Republishes every binding at 1.0.2 to close the 1.0.1 gaps (notably PHP/Packagist).
- #1321: Borderless, text-heavy tables are recovered on pages that also contain an ML-detected
table. The geometric-table fallback now runs per region instead of per page, so a single ML
Tablehint no longer suppresses borderless-grid recovery across the rest of the page; words already inside an existing table hint are excluded so regions are not detected twice. - #1326: RTF hex byte escapes now decode through the active font's
\fcharsetNcharset (mapped to a Windows codepage), falling back to\ansicpgNNNNand then Windows-1252. Documents that declare a Cyrillic or other non-ANSI font in the font table now decode as readable text instead of Windows-1252 mojibake, and font switches mid-document are tracked across nested groups. - #1328: Page markers now appear verbatim in Markdown and Djot output. Flat documents no longer
backslash-escape the marker (
\<\!-- PAGE 1 --\>), and structured native documents no longer drop it entirely. - #1323: RTF hex byte escapes now honor
\ansicpgNNNNvia the shared Windows-codepage table, so CP1251 Cyrillic and other non-1252 ANSI byte runs decode as readable text instead of Windows-1252 mojibake; adjacent escapes decode as one multi-byte run, surviving line wraps, and formatting spans stay aligned with the decoded text.
LayoutStrategyenum onLayoutDetectionConfig(strategyfield, defaultalways).autopre-screens each PDF page with cheap geometry signals and runs the layout model only on pages likely to benefit; existing configs keep the every-page behavior bit-for-bit. On the OCR path only inference is skipped, since OCR consumes the layout pass's rasters. Skipped pages are auditable viametadata.format.layout_gated_pagesandlayout_gate_reasons, and the CLI gains--layout-strategy(#1322).
- Republishes
xberg-libwpdwith the static zlib link fix soxberg-clilinks against a working release. The 1.0.0xberg-libwpdcrate was published before the fix and left the librevengeinflateInit2_/inflate/inflateEndsymbols undefined at final link, breakingxberg-clibuilds from crates.io. No source API changes.
xberg 1.0.0 is the first stable release of the document-intelligence engine previously developed as Kreuzberg. It is the direct successor to Kreuzberg v4.9 and carries the same Rust core and extraction-API lineage forward under the xberg name. The Kreuzberg v4 line continues as LTS at kreuzberg-dev/kreuzberg-lts. This entry summarizes everything that changed relative to Kreuzberg v4.9.
Beyond the rename, 1.0.0 is a large release: the PDF stack moved to a pure-Rust backend, the OCR story grew from a single engine to a family of classical and vision-language models, and whole new capabilities landed — audio/video transcription, named-entity recognition, structured LLM extraction, sparse/late-interaction retrieval, and four new language bindings.
For a step-by-step upgrade, see the migration guide.
- Packages are renamed
kreuzberg→xbergacross every ecosystem (crates.io, PyPI, npm, Maven, NuGet, Composer, RubyGems, Hex, Go). - The Rust error type
KreuzbergErroris nowXbergError. - Environment variables are re-prefixed
KREUZBERG_*→XBERG_*, and config files are discovered asxberg.{toml,yaml,yml,json}. - Breaking API changes: extracted URIs are returned as
ExtractedUri(formerlyUri); document metadata drops the untypedadditional/serde-flatten bag in favour of typed fields plus acustomresidual map. - The R binding, the EasyOCR backend, and the bundled pdfium fork are removed (see Removed). Existing Kreuzberg v4 installs keep working under their original names.
The full identifier mapping is in the migration guide.
- A family of OCR backends. Alongside Tesseract, 1.0.0 adds a native PaddleOCR backend
(PP-OCRv6, with
medium/small/tinytiers) and a pure-Rust Candle OCR/VLM stack — TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL — that runs without ONNX Runtime or native Tesseract. Model weights are self-hosted on thexberg-ioHugging Face org. - A second, ONNX-Runtime-free inference path (tract). CNN classifiers, layout detection (RT-DETR),
and auto-rotation run through a pure-Rust
tractbackend on targets without ONNX Runtime — this is what makes in-browser (WASM) and mobile inference possible. - Structured (LLM) extraction.
extract_structuredandsplit_and_extractdrive a vision-LLM client with rasterization, chunking, citations, caching, and configurableCallMode/MergeMode/ VLM-fallback policies. - Audio and video transcription. A Whisper ONNX encoder/decoder engine extracts text from
.mp3,.wav,.m4a,.mp4, and.webm. - Named-entity recognition. GLiNER2-based entity extraction, including an in-browser WASM
NerModelthat detects entities locally with no server round-trip. - Retrieval building blocks. Sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and a cross-encoder reranking / semantic-search stage alongside dense embeddings, with self-hosted model presets pinned by sha256 manifests.
- Text intelligence. Redaction with reversible rehydration and per-entity erasure, summarization,
translation, VLM image captioning, QR-code detection, document diffing (
revisionsonExtractionResult), and page/chunk classification. - URL and web ingestion.
map_urldiscovers URLs from sitemaps and a shared crawl engine batches multi-URL extraction, over a URI-basedExtractInput/ExtractionOutputenvelope. - New document formats (98 total). WordPerfect
.wpd/.wp/.wp5(via a vendoredxberg-libwpd), HEIC/HEIF/AVIF images (via a vendored libheif), OpenDocument Presentation.odp, Quarto/R Markdown, configurable Jupyter cell rendering, and the audio/video formats above. - Four new language bindings. Dart/Flutter, Swift, Kotlin/Android, and Zig — for 15 language bindings over one engine, with Android/iOS cross-compilation.
- First-party integrations, consolidated into the monorepo. LangChain.js, LlamaIndex, an n8n community node, CrewAI, and a Spring AI document reader.
- Richer chunking and API surface. Caller-supplied tokenizers,
TableChunkingMode::RepeatHeader, RAG chunking with heading-path breadcrumbs, multi-label chunk classification, per-page spans with bounding boxes, alist_supported_formats()call in every binding, cheappdf_page_count, and aDELETE /jobs/{job_id}cancellation endpoint on the API server. - Wider code intelligence. tree-sitter coverage grows from 248 to 306 programming languages.
- PDF backend replaced. pdfium is gone;
pdf_oxide, a pure-Rust engine, is now the sole PDF backend — no native pdfium dependency. - Layout-aware PDF pipeline. Reading order is reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering; scanned PDFs are detected and OCR'd selectively per page; AcroForm/XFA form fields and outline-based headings are extracted.
- Public API stabilized and frozen for 1.0, with a Rust-only
Engineand extension seams. - Renamed from Kreuzberg to xberg across packages, namespaces, and the
KreuzbergError→XbergErrortype (see Migration). - Environment variables use the
XBERG_prefix; new layout, OCR model-tier, CoreML, and ORT execution-provider variables are available. - Config discovery now also accepts the
.ymlextension (xberg.{toml,yaml,yml,json}) with an XDG config-directory fallback. - Models and cache live under the
xbergcache segment and thexberg-ioHugging Face org; the project domain isxberg.io. - Python support widens to 3.10–3.14; the SurrealDB connector moves to v3 (dropping
mem://); the defaultextraction_timeout_secsis 60s. - License. Relative to the Kreuzberg 4.8/4.9 line (Elastic License 2.0), xberg 1.0.0 is MIT.
More than 150 bugs were resolved during the 1.0 cycle. Highlights by area:
-
PDF text fidelity: text inside Marked-Content (MCID) blocks is no longer dropped from markdown/HTML output (#917); ligature glyphs no longer map to control characters (#1135); glyph-spaced text no longer extracts one character per line (#962); spurious intra-word spaces in native extraction are fixed (#1291, #1222); JPEG 2000 images no longer render blank and silently break OCR (#1158); XML entity references (
&/</>) are preserved (#1242). -
Tables: bordered / graphical-line tables are detected reliably instead of silently skipped (#964, #1097, #1213); rotated full-page tables no longer extract as word salad (#1220, #1221); duplicate table emission and double-counting are fixed (#1288); physically fragmented per-row tables are merged back with their header row (#1290, #1100); borderless and text-heavy grids keep their row associations (#1316, #1319).
-
Reading order & structure: two-column reading order no longer scrambles headings (#1170); stale page boundaries after reordering no longer panic on multibyte text or drop documents (#1270, #1272); numbered and cover-page headings are classified correctly (#961, #966, #1096, #1098); filled form field values are placed correctly (#1120).
-
OCR: an explicit PaddleOCR backend no longer silently falls back to Tesseract (#801, #1071, #1088, #1102); scanned-page OCR text and page provenance surface consistently across content, pages, and chunks (#1095, #1110, #1281); spurious auto-OCR on born-digital PDFs is suppressed (#1176); a SIGBUS crash and a NaN-sort panic in the OCR pipeline are fixed (#1057, #1179); the Candle VLM OCR backends are stabilized (#1174, #1175, #1208–#1214); model downloads handle TLS-MITM CAs, IPv6 blackholes, and connect timeouts (#1146, #1249).
-
Chunking & provenance: chunk
firstPage/lastPageand byte ranges are correct across output formats and long PDFs (#1013, #1074, #1105, #1294); markdown chunks retain markdown (#1073, #1094); split-table chunks keep their header and context (#1100). -
Bindings & packaging: fixed Go embed symbols, Java
UnsatisfiedLinkError, missing C# config types, wrong Node/PHP embedding shapes, Android.soloading, macOS wheel floors, and musl/ONNX Runtime runtime deps (#871, #965, #991, #998, #1008, #1055, #1131, #1257, #1304, #1307); plus Homebrew 404s, Docker stop-signal handling, and multi-arch-coreimages (#1081, #1147, #1247, #1315). -
Formats: EML HTML
<table>bodies, DOCX hyperlink/bold overlap and markdown conversion, archived markdown/CSV escaping, and Korean-charset EML detection are fixed (#942, #1086, #1212, #1237, #1278). -
Config & robustness:
extraction_timeout_secsis honoured on every path (#830, #911, #1273);cancel_token, custom LLM base URLs, and page-classification config all validate correctly (#937,#944, #1076).
- R binding — the Kreuzberg v4 LTS line is the last to ship it.
- EasyOCR backend — the Python/torch-only backend did not survive the Rust rewrite; use Tesseract, PaddleOCR, a Candle backend, or a VLM backend instead.
- Hunyuan-OCR Candle backend — ported during development, then dropped before 1.0.0.
- Bundled
pdfium-renderfork and itsKREUZBERG_PDFIUM_BUNDLED_PATHvariable, and the standalone@kreuzberg/corenpm package.
- OCR memory discipline: concurrent Tesseract sessions are capped, Leptonica/Pix/page buffers are released early, decoded RGB buffers are reused, and images are resized without copies.
- Layout inference: model sessions are pooled, batch inference threads are balanced, and an unnecessary PNG raster round-trip is bypassed.
- Engine and batch: bounded batch scheduling, no per-item config clone, single PDF parse per structured rasterization, streamed batch JSON output, and base64 hosted embeddings.
- PDF and text: streamed RGB conversion, reused OCR render document, skipped redundant compatibility parses, and a regex→scanner rewrite that removes backtracking from text/quality cleanup.
-
An untrusted RTF size field in the email extractor could allocate up to 4 GB — now bounded (#1058).
-
A redaction path could leak PII across roughly a dozen output fields — fixed (#1223).
-
PDF embedded streams are guarded by a decompression-ratio limit and per-embedded-file size caps, and a
SecurityBudgetis wired through the PDF and email extractors. -
Excel DDE / external-call formulas raise warnings during extraction.
-
FFI image and attachment buffers now carry explicit lengths so callees never read past the buffer (#1056, #1059), and panics on malformed input are replaced with recoverable errors (#907, #1057,
#1198).
- pdf_oxide replaces the pdfium native dependency; libheif (LGPL, documented) and
xberg-libwpdare vendored; retrieval and OCR model presets are self-hosted onxberg-iowith sha256 manifests. - Distribution hardening across all 15 targets: ONNX Runtime bundling, glibc/musl floors (musl via
Alpine images), NuGet runtime-package size splits, Homebrew bottles, Go module tags, Swift C++
linkage, and Dart/Swift/Kotlin-Android/Zig release matrices. Published to crates.io, PyPI, npm,
Maven Central, NuGet, RubyGems, Packagist, Hex, pub.dev, Go, Swift Package Manager, Homebrew, Docker
(
ghcr.io/xberg-io/xberg), and a Helm chart.