Skip to content

Releases: xberg-io/xberg

v1.2.7

Choose a tag to compare

@Goldziher Goldziher released this 22 Sep 16:42
v1.2.7
fa4eeb0

Added

  • (config): ConcurrencyConfig::max_concurrent_ocr and the --max-concurrent-ocr CLI flag set concurrent Tesseract recognition sessions on their own. Use it when the host has cores to spare but not the memory to run a recognition session on each of them. The value is applied as given: neither the thread budget nor the host's free memory reduces it, since both of those bound only the automatic limit. The first extraction in a process fixes the session count for the rest of that process, because the admission semaphore and the Tesseract handle pool that enforce it are built once and the pool's capacity is fixed when it is constructed; a later extraction that names a different value keeps the first one and logs a one-time WARN naming both numbers. Set the value on the first extraction, or run one process per value. ConcurrencyConfig is not #[non_exhaustive], so the added field breaks any Rust caller that builds the struct by literal without ..Default::default(); such a caller must add the field or the rest pattern. Callers on every other binding are unaffected. (GH#1727)

Changed

  • (pdf): scan detection and fabricated-mapping OCR routing now use the thread budget. Native PDF extraction graded every page for raster scan evidence, then read every page's text again to check its glyph-to-Unicode mapping provenance, and ran both passes one page at a time. The two passes held a single core for the whole document whatever ConcurrencyConfig::max_threads was set to, so no thread budget could shorten them. Both now run across the thread budget, sequentially on wasm32 which has no thread pool, and report the same per-page confidences and the same page lists in the same page order. Measured on a 32-core machine at max_threads = 32, with the extracted output identical in every run: a 4778-page manual from 66.9 s to 63.3 s, and a 676-page book from 24.1 s to 21.9 s. The two passes are a small share of a native extraction, so the wall-clock gain is bounded by that share. (GH#1723)
  • (ocr): concurrent Tesseract recognition follows the thread budget instead of a fixed four. The limit was a compile-time constant that no configuration reached, so raising max_threads could not raise recognition throughput, and recognition is most of the run on a scanned document. The default is now the thread budget, reduced to the number of sessions the host's free memory holds. Set max_concurrent_ocr to 4 to keep the previous behaviour. (GH#1727)
  • (pdf/ocr): the per-page OCR route sizes its batch by what a page costs to hold, not by the limit on returned content size. max_content_size bounds the text an extraction returns and says nothing about the rasters a batch holds while it works, so on an ordinary 150 DPI document the arithmetic pinned the batch at four pages and no stage of the route could use more than four threads whatever the thread budget said. The width now comes from the page box and the render resolution, measured against free memory less the document and a reserve. Each page's own peak is still checked against max_content_size, one page at a time. Peak resident memory rises with the wider batch -- 2,234 MB to 3,583 MB on a 731-page document at a 32-thread budget -- because more rasters are in flight. (GH#1724)
  • (pdf): the embedded-image pass now runs across the thread budget. Configuring OCR switches whole-document embedded-image extraction on, and that pass walked the document one page at a time, so no thread budget shortened it. Pages now decode in parallel (sequentially on wasm32, which has no thread pool), in the same order, and the returned bytes are unchanged. Measured on a 731-page document holding 2,881 images at a budget of 32: the pass falls from 20.1 s to 2.2 s, and the whole extraction from 57.3 s to 41.3 s. Peak resident memory rises with the budget, because each pool thread holds one page's raw pixel buffers and their PNG re-encodes at once, where a single page's were live before: over the pass itself, 1,133 MB before against 1,463 MB at a budget of 8 and 2,186 MB at 32. The pass carries no memory budget of its own, so the ceiling is the thread budget, which is min(cpu_cores, 8) until ConcurrencyConfig::max_threads raises it. The pass also no longer decodes and PNG re-encodes a full-page image on a page that already carries native text when OCR is the only reason the pass is running at all: should_skip_pdf_image_ocr excludes exactly that image from OCR and drop_ocr_only_images then drops it from the result unread, so the decode was pure waste. That image still appears in the result with its dimensions, bounding box, page number and alt text intact and an empty data; a document where any other consumer -- image extraction, captioning, QR codes, inline-image OCR, page rasters, or styled HTML rendering -- would actually read the bytes is unaffected. (GH#1732)
  • (ocr): candle-deepseek-ocr reads device tensors back in one transfer instead of one element at a time. The mixture-of-experts gate copied its score matrix per expert, per row, per layer, per generated token, and the SAM relative-position lookup read its index inside a nested loop per attention layer, per crop. Both are now a single bulk read. Measured on an A100 at BF16: 62.71 s to 8.09 s per page (7.8x), output byte-identical, peak memory unchanged. A 22-page scanned document that previously exceeded the 600 s extraction ceiling now completes in 322 s. (GH#1711, GH#1714)
  • (ocr): candle-deepseek-ocr resizes the relative-position table and scatters the image embeddings as device operations instead of per-element loops. The resize sampled its axis one position at a time and concatenated one tensor per output position, on each of the four global-attention layers of the batched 640 px local-crop pass; it is now candle's bilinear resize over a one-row image, which is the half-pixel sampling the reference implementation uses. The scatter gathered rows one at a time and stacked one tensor per sequence position, once per page; it is now two gathers and a select. Measured on an NVIDIA L4 at BF16 over a full Letter page, five repetitions: 47.610 s to 47.546 s per page, which sits inside the spread of the unchanged arm, so this is not a speedup on that workload. Both arms decode the same 2,052 characters. Peak GPU memory falls from 8,818 MiB to 8,722 MiB. (GH#1719)
  • (ocr): the per-page OCR-image render/encode size check now runs through one shared helper instead of three separate copies. GH#1724 and GH#1731 each fixed a route that summed a whole page batch against security_limits.max_content_size -- a limit its own error text names as a per-image bound -- rejecting every page in the batch once the sum crossed it. Both fixes, and the mixed native/OCR route's own single-backend path, now call one validate_png_encode_pages_individually helper that charges each page on its own, so a new call site cannot reintroduce the batch-summing shape by calling the lower-level batch-peak function directly. No behaviour change on any of the three existing routes. (GH#1748)
  • (pdf): a /Font dictionary shared across pages no longer redoes TrueType cmap donation on every page that touches it. share_truetype_cmaps clones and mutates each undonated font with Arc::make_mut, and the font-set caches deliberately store each dictionary's pre-donation fonts (#1725 made that order-independent), so every cache hit re-ran that clone for the same font. Donation among fonts within one dictionary's own resolved set is now cached alongside the pre-donation set, under the same key, so it depends only on that dictionary's own resources and not on which page or Form XObject was read before it. Donation whose donor lives in a different dictionary (a Form XObject donating to its page, say) is unaffected and still runs on every load_fonts call. (GH#1746)
  • (pdf): fabricated-mapping OCR routing no longer reads every page's text a second time. The main text pass and the provenance pass each read every page of a native PDF: the main pass to assemble content, the provenance pass (GH#1723) to grade each page's glyph-to-Unicode mapping. Neither carried its read into the other, so a 731-page document spent 14.2 s of a 24.6 s extraction on the two passes together. The main pass now keeps each page's fabricated-character counts from the raw spans it already reads, and the provenance pass consumes those counts instead of reading the page again; a document with default-off optional-content layers still reads separately, since the main pass reads layer-filtered spans there and the two would otherwise disagree. Retaining the counts holds page text already held for content assembly, not rasters, so peak memory is unaffected. fabricated_text_pages, scan confidences and extracted content are unchanged. (GH#1744)
  • (pdf): scan detection no longer decodes an embedded image's pixels just to classify its codec. Grading a page for raster scan evidence decoded every embedded image on the page in full -- the same decode the embedded-image extraction pass repeats moments later when OCR is configured -- only to read whether the result was JPEG or CCITT and how large its bounding box was. Both are already known from the cheap Phase 1 handle enumeration that walks the content stream without decompressing anything, so scan detection now reads bounding box and filter chain from there instead. The embedded-image extraction pass, the only remaining caller that needs decoded pixels, is unaffected; classification output (image area ratio, codec class) is unchanged for every image that decodes; an image the decoder rejects now counts toward the page's image coverage where it was silently dropped before, and images under the 8 x 8 px extraction floor stay excluded. (GH#1732)

Fixed

  • (pdf): layout detection no longer fails partway through a long document. The layout pass charged every page raster it had already produced aga...
Read more

v1.2.6

Choose a tag to compare

@Goldziher Goldziher released this 20 Sep 09:51

Added

  • (pdf/ocr): OcrQualityThresholds::enable_plausibility_ocr_routing and OcrQualityThresholds::min_reliable_language_chunk_ratio control language/dictionary-plausibility OCR routing, and PdfMetadata.implausible_text_pages reports which pages a page's decoded text failed to read as any real language. (GH#1696)
  • (config): ExtractionConfig::runs_ocr_on_embedded_images and ExtractionConfig::wants_own_bytes_in_result are exposed on every binding, beside the existing needs_image_data. The first is the predicate the pipeline uses to decide whether a container's embedded images are OCR'd; the second is the pre-GH#1662 formula (extract_images, captioning or QR codes) that decides whether a standalone image's own bytes are echoed into images. (GH#1662)

Changed

  • (ocr): the ocr, ocr-wasm, and ocr-pipeline Cargo features now imply language-detection. The language/dictionary-plausibility OCR-routing signal (GH#1696) is gated the same as the rest of the OCR module and would otherwise be a silent hole under a build that enables only ocr-pipeline (the VLM-only, Tesseract-free pipeline). whatlang (behind language-detection) depends only on hashbrown, so this is wasm/Android-safe. (GH#1696)

Fixed

  • (ocr): OCR no longer returns every embedded image's raw bytes. Image bytes read only to feed embedded-image OCR (GH#1662) are dropped once OCR has consumed them and any OCR text it produced has been rendered into content; images now carries data only when the caller asked for image extraction, captioning, QR-code detection, pdf_options.ocr_inline_images, or images.include_page_rasters; counts.images still reports how many embedded images the document had. (GH#1703)
  • (pdf): a /ToUnicode mapping that resolves to the wrong-but-real letters is now routed to OCR too. GH#1667 fixed the case where a page has no usable mapping tier at all (MappingProvenance::Fallback); a page whose /ToUnicode CMap DOES resolve, but consistently to the wrong letter (e.g. a ROT-shifted mapping), passed every character-shape check unchanged and OCR never fired, because the text is structurally indistinguishable from real prose. A language/dictionary-plausibility signal now flags such a page under the default Auto strategy, and the opt-in ScannedPages strategy picks it up through the existing scanned_pages union. A page with no explicit ocr config keeps its native text, but a warning now names the defect. Closes the remaining gap in GH#1667. (GH#1696)
  • (pdf): the mixed native-and-OCR path no longer drops every native table on a page OCR never touched. On a long document where only some pages needed OCR, the mixed path replaced the whole document's table list with just the OCR pages' tables whenever OCR found even one, silently dropping every native table on every other page. Only the tables of pages OCR itself produced a table for are now replaced; a page the mixed path never sent to OCR keeps its native tables. (GH#1670)
  • (pdf): include_document_structure alone now triggers the structured native extraction pass. Previously the structured pass ran only for a Markdown/Djot/HTML/DocTags output format, an explicit hierarchy config, inline-image OCR, or a content filter -- never for include_document_structure itself. A caller who set only that flag, with output left at its Plain default, silently got a flat, paragraph-only document, so the structure tree it asked for came back holding nothing but paragraph nodes even though the extractor found headings, tables and page breaks. This configuration now renders its plain-text output from the structured document, so the rendered content changes with it: table rows render one per line, and headings and page footers reflow. A caller that sets only this flag and depends on the previous Plain text sees that text change. The table count is unaffected. (GH#1668)
  • (docx): a paragraph inherits the list numbering that its style carries. A list whose paragraphs hold no direct numbering and take it from the paragraph style instead, which is the shape Word writes for its built-in list styles, came out as plain paragraphs with no list markers. The parser now resolves numbering through the style's basedOn chain. Direct paragraph numbering still takes precedence over the style. (GH#1663)
  • (pdf): the fabricated-glyph-mapping provenance lookup is repaired, and a page it flags is sent to OCR. The extractor read each text run's mapping provenance after it had renamed the run's font to the resolved /BaseFont, but the provenance table is keyed by the raw resource alias, so the lookup missed for every font whose base name differs from its alias and the signal was empty almost everywhere. The lookup now runs in the same pass and uses the alias, and a page whose text carries no usable mapping tier at all (MappingProvenance::Fallback) now forces the OCR fallback under the default Auto strategy. A font whose mapping resolves, even to the wrong characters, is a different failure mode this signal does not cover. (GH#1667)
  • (ocr): a low OCR confidence now lowers the reported quality score. The quality score measured only how clean the text looked, so a page that OCR itself had little confidence in still scored as clean whenever the characters happened to be well formed. The score is now capped by the word-count-weighted mean of per-page OCR confidence, once at least 20 words were recognized -- the same fold ExtractionConfidence::ocr_aggregate uses, shared rather than recomputed, though the aggregate itself reports on any recognized word and so can differ below the 20-word floor. A native, non-OCR extraction is unaffected. (GH#1669, GH#1694)
  • (ocr): a cold start no longer builds one candle OCR engine per waiting caller. Each candle backend's engine pool looked for a cached engine, released the lock, built the engine, then stored it, so every request that arrived during the first build loaded its own copy of the model and a burst could exhaust the GPU memory. All four backends and the GLM-OCR layout pool now use the crate's engine cache, one per backend, which holds the write lock across the build: the first caller builds and the rest wait for that engine. Loads of different engines on the same backend (two TrOCR variants, say) therefore also serialise during a cold start. A failed load stores nothing, so the next caller retries. (GH#1683)
  • (ocr): the aggregate OCR confidence is populated for page-level results, and both routes weight it the same way. A document whose OCR confidence arrived one score per page left the aggregate empty, so a caller that read it got nothing back. Both routes now fold the scores by recognized word count instead of by element, so a page of many short lines no longer counts for as much as a page of few long ones. This changes the number the embedded-image route already reported. (GH#1677)
  • (ocr): the Tesseract language probe run at backend registration now finds the installed languages. The probe initialised Tesseract with an empty datapath, which makes the library look one tessdata directory deeper than the resolver real OCR jobs use, so on a layout where jobs found eng.traineddata directly the probe logged "couldn't load any languages" and silently fell back to a hardcoded language list. It now resolves the tessdata directory the same way a job does. (GH#1671)
  • (ocr): a page whose embedded-image OCR retry came back empty says so. When page-raster OCR failed and the embedded-image retry ran but returned nothing -- a candle VLM backend reports success with empty content -- the page warning named only the first failure, with no trace that a retry ran at all. The per-page warning and the wholesale "every page failed" error now report the retry outcome alongside the first failure. (GH#1673)
  • (ocr): OCR throughput scales with the thread budget. The mixed native-and-OCR route rasterised each batch's pages one at a time regardless of the configured thread budget, so a 240-page document with 80 scanned pages took the same 9 s at 4, 8, 16 and 32 threads. Pages now render in parallel across the batch (sequentially on wasm32, which has no thread pool), in the same order as before; measured 8.8 s to 3.3 s at 4 threads and 9.0 s to 1.0 s at 32. (GH#1666)
  • (ocr): a DOCX, PPT, PPTX or HTML embedded image is read before OCR runs on it. The predicate that decides whether a container reads an image out of its archive did not count embedded-image OCR, so with OCR configured and run_ocr_on_images left on, the image was attached with an empty buffer and OCR reported "Could not determine image format" for zero bytes. A standalone image extraction's images output is unchanged. (GH#1662)
  • (ocr): the mixed native-and-OCR route sizes its render batch against security_limits.max_content_size. The batch was sized from the resolved thread budget alone, so a wide thread budget requested a batch whose estimated PNG-encode peak crossed the fixed content limit, and every page in the rejected batch was skipped rather than the extraction failing. The batch is now capped up front from one representative page's estimated encode cost at the effective render DPI; the real peak is still checked, and still rejected if genuinely too large, afterwards. (GH#1665)
  • (ocr): the candle OCR backends no longer reject every rendered PDF page. The PDF OCR route stamps source_dpi and page_rotation_degrees into the shared backend_options for every backend, and the candle backends (candle-paddleocr-vl, candle-glm-ocr, candle-deepseek-ocr) deserialise their options strictly, so every rendered page failed validation and came back empty. Exactly those two pipeline hint keys are stripped before the candle options are read; any other unknown key still fails validation. (GH#1672)
  • (ruby): a tagged enum's payload readers return the stored value. The generated Data reader...
Read more

v1.2.5

Choose a tag to compare

@Goldziher Goldziher released this 18 Sep 18:43

1.2.4 was tagged but never reached a package registry: its generated binding files still
named 1.2.3, so its publish run was cancelled. 1.2.5 is the first published release carrying
the 1.2.4 changes below in addition to its own.

Fixed

  • (python): a config field holding a nested data-carrying enum is no longer dropped by
    from_json / to_json.
    The Python DTOs marked every field whose type transitively held
    a data enum as #[serde(skip)] -- CaptioningConfig.llm and ChunkingConfig's nested
    options among them -- so a JSON payload round-tripped through such a config silently lost
    those fields. Regenerated on alef 0.92.1, whose Python emitter no longer treats a
    serializable enum wrapper as opaque. (alef#394)
  • (ppt): a legacy .ppt's embedded OLE objects are extracted. A Word document or Excel
    sheet inserted as an object -- the way PowerPoint 97-2003 decks routinely carry a table --
    lives in the deck's ExOleObjStg records, which the extractor walked over as opaque bytes,
    so the slide came out as its title and nothing else. The .pptx path already recursed into
    ppt/embeddings/; the legacy path now does the same: each embedded object's storage is
    resolved through the persist chain (so a superseded save's copy is never read),
    decompressed when the record says so, identified by its root stream (WordDocument,
    Workbook/Book, PowerPoint Document, or an OPC Package) and extracted into
    children as slide<N>/oleObject<id>.bin with the slide whose shape displays it. The
    decompressed size is bounded by max_embedded_file_bytes (falling back to
    max_archive_size), the object count by max_files_in_archive, and max_archive_depth = 0
    disables the recursion, as for .pptx. Metafile presentation pictures (EMF/WMF) are still
    not emitted as images. (GH#1660)
  • (pdf): a ruled table drawn one bar per cell no longer loses all but its last two rows.
    Word draws cell borders as per-cell filled bars with a corner square at every crossing, so
    every vertical rule terminates at every horizontal rule. The section-divider split, which
    reads a full-width rule that no vertical rule runs through as the boundary between two
    stacked form sections, therefore treated every internal rule of such a table as a divider,
    cut the table into one-row pieces and discarded the pieces below the cell minimum -- a
    12-row table came back as 11 | 290, 12 | 300. A rule that every vertical rule stops at
    carries no section information when the same is true of every other rule on the table, so
    a table whose internal rules all qualify is now left whole; a form whose sections' rules
    stop at a minority of the rules still splits. (GH#1656)

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.2.5/xberg-zig-v1.2.5.tar.gz\",\n        .hash = \"xberg-1.2.5-iV1GrqrLSBkt-vzYK25PD31qnEAwcTdEd0EFd3EG4Flm\",\n    },\n},\n```\n

v1.2.4

Choose a tag to compare

@Goldziher Goldziher released this 18 Sep 13:24
v1.2.4
68231ee

Not published to package registries. The generated binding files in this tag still name 1.2.3 (the Dart loader would fetch the v1.2.3 native), so its Publish Release run was cancelled before any registry upload. Superseded by v1.2.5, which carries the same fixes plus GH#1656 (segmented rules) and GH#1660. Docker images for 1.2.4 are self-contained and unaffected.

Added

  • (llm): LlmConfig exposes liter-llm's provider response-byte cap. liter-llm's
    ClientConfigBuilder::max_response_bytes bounds every HTTP response body read from a
    provider, but xberg's LlmConfig never forwarded it, so downstream products had no way to
    bound provider responses. LlmConfig::max_response_bytes mirrors it: it bounds response
    bodies on every non-streaming call and the error body read on a failed request (a
    successful streaming response keeps its own existing frame bounds), and defaults to None
    (unbounded), matching liter-llm. Some(0) is rejected by LlmConfig::validate rather than
    reaching liter-llm's own builder. (xberg-io/xberg-enterprise#1568, xberg-io/xberg-enterprise#1861)

Fixed

  • (pdf): a page-sized background rectangle no longer seeds a whole-page table. Many
    Office-to-PDF producers draw a white rectangle over the entire page before anything else.
    That rectangle passed the table-primitive filter (its < 1000 pt bound does not exclude an
    A4 or Letter page) and, because clustering unions any two primitives whose boxes intersect,
    pulled every rule on the page -- the real table's borders, the footer rule, a figure's
    frame, strokes inside a drawing -- into one cluster with the page as its box. The cluster
    fallback then built a table over the whole page: the heading above the real table became
    its first row, torn at the table's own column rule, and the first words of the prose below
    it became its last row, with those words missing from or doubled in the paragraphs that
    followed. A rectangle covering at least 90 % of the page's MediaBox in both dimensions is now
    treated as page furniture and dropped before clustering, so the real table's rules cluster
    on their own and the page comes out as it does without the background. A full-page-width
    rule (one dimension at page scale) is deliberately still a primitive. (GH#1656)
  • (pdf): a single-column page of hanging-number headings is no longer read as two columns.
    Some producers (Distiller among them) emit the tab between a heading's number and its title
    as a space-only span in a different font, and leave space-only spans for blank lines. The
    dense-two-column repair counted those spans as column population and let a blank line's two
    whitespace spans and a footer split across both margins vote for a gutter, so five headings'
    tabs plus two lines of nothing met the six-line quorum and the page was emitted
    column-major: 6, 6.1, 6.1.1 as bare numbers and their titles as separate unnumbered
    headings. Whitespace-only spans no longer count toward the per-side density gate, the
    gutter vote, the whitespace corridors or the row-pairing guard, and a per-line gap wider
    than a quarter of the page width is no longer accepted as gutter evidence (a blank line's or
    a footer's gap; every real gutter measured here is under 10 %). (GH#1655)
  • (ocr): a configured security_limits now reaches the Tesseract image decode. GH#1554
    routed ExtractionConfig::security_limits onto OcrConfig before each OCR call, but the
    Tesseract backend's config_to_tesseract had no field to carry it into and built a fresh
    default ExtractionConfig for the processor, so every Tesseract decode -- standalone
    images, embedded images, scanned pages and the targeted page fallback alike -- ran under
    the 100 MiB default no matter what the caller set. A 600 DPI A4 scan was refused with an
    error naming 104857600 bytes while both configuration objects said 5 GiB. The internal
    TesseractConfig now carries the limits (outside the cache key: they gate whether a decode
    runs, never what it produces), the processor prefers them over its synthetic config, and
    the standalone-image and PDF embedded-image routes inject the caller's limits like the
    other routes already did. A limit set directly on OcrConfig is honoured when
    ExtractionConfig carries none; ExtractionConfig still wins when both are set. (GH#1651)
  • (go): Register* no longer leaks its cgo.Handle when the C vtable allocation fails.
    Every Register* wrapper in the Go binding created the handle before allocating the C
    vtable, and the allocation-failure branch returned without deleting it, keeping the bridge
    and the caller's implementation reachable for the life of the process. The branch is only
    reachable when a small malloc fails, so this is a resource leak on an error path rather
    than an exploitable defect; it is fixed in the generator (alef 0.91.6) and regenerated
    here. Reported by @OvOhao (GHSA-q5pq-8g86-v9j9).
  • a table's multi-word cell no longer bleeds a trailing word into the next column, a
    right-aligned amount column split by digit-width drift is folded back together, and a
    legitimately sparse but independently-headed column (or a sparse first data row) no longer gets
    the whole table rejected.
    Reconstructing a table from word boxes decided each word's column
    independently by nearest x-position, so a wide cell's own trailing word (e.g. a long
    description's last word) could resolve to a neighboring column's anchor instead of its own
    cell's. Column membership for a data row is now decided once per merged cell cluster and applied
    to every word the cluster contains; a drift-split right-aligned numeric column (mutually
    exclusive per row, no independent header of its own) is folded into one column; and a section
    caption sharing a table region with its real header row is dropped. This cell-clustering and
    column-assignment logic lives in the shared table_core module, so it also affects native-PDF
    word-position table reconstruction, not just OCR. Separately, pdf's shared table post-processor
    no longer treats an independently-headed but infrequently-populated column (e.g. a bank
    statement's DEPOSIT, populated on a minority of transaction rows) as noise, and no longer folds
    a sparse first data row (one missing an optional numeric field) into a bogus multi-row header
    merge with the row after it -- and that header-shortcut no longer stops one row early on a
    genuine two-row text header, which also fully populates a digit-free first row. The punctuation
    cell-merge gap this widens for a lone dash/colon glyph is now conditioned on the punctuation
    genuinely bridging two neighboring words on both sides, so an isolated punctuation cell near a
    real column boundary (a - placeholder, a :) no longer fuses two columns' content into one
    cell. (GH#1649)

v1.2.3

Choose a tag to compare

@Goldziher Goldziher released this 16 Sep 20:19

Added

  • (ocr): the registry reports each backend's declared languages. list_ocr_backends() returned
    only names, so a caller validating an OCR language against the installed backends had to maintain
    a second language table of its own. list_ocr_backend_capabilities() returns one
    OcrBackendCapabilities record per registered backend — its name and the languages it reports —
    ordered by backend name. An empty supported_languages means the backend does not enumerate its
    languages, not that it supports none, so ocr_backend_supports_language() is provided for
    callers that need a decision rather than a list. (GH#1643)
  • (ocr): mixed-page PDF extraction reports one coordinate frame per OCR'd page. Public
    ocr_elements carry the OCR backend's raster coordinates, but the per-page processed raster size
    lived only in the page-local result and was discarded before the elements reached the final
    document. A multi-page consumer was left with the document-level
    ocr_processed_image_width/ocr_processed_image_height, which is not authoritative for pages of
    differing size, preprocessing, or rotation, and so could not normalize a box safely.
    metadata.additional.ocr_page_coordinate_frames now carries one record per OCR page that has
    public elements — page_number, width, height, unit: "pixel", origin: "top_left" — ordered
    by page number and joined to an element through the element's own page_number. No record is
    emitted for a page whose raster dimensions are absent or invalid. (GH#1645)
  • (ner): Rust callers can share xberg's process-wide GLiNER backend cache.
    text::ner::gline::get_or_init_backend performs model initialization on Tokio's blocking pool,
    returns the same Arc for the same model and thread budget, and leaves failed initializations
    retryable.
  • (pdf): every hierarchy page now advertises the coordinate frame its boxes live in. Hierarchy
    bounding boxes are emitted in raw PDF user space, but nothing named that space, so a consumer
    could not tell whether a box needed translating by a non-zero MediaBox origin or rotating for a
    page's /Rotate. metadata.additional.pdf_page_coordinate_frames now carries one record per
    page that has hierarchy blocks — page_number, origin_x/origin_y (the MediaBox llx/lly,
    which may be non-zero and negative), width/height, unit: "point",
    origin: "bottom_left", and clockwise_rotation — ordered by page number. width/height are
    the MediaBox extent and are deliberately not swapped for a 90/270 rotation, because they
    describe raw user space rather than the displayed frame; this is the opposite convention to
    ocr_page_coordinate_frames, which reports an already-rotated raster, so the two must not be
    assumed to agree for the same page. A page is omitted entirely, rather than reported with a
    default, when its /Rotate is present but not a valid multiple of 90 or its MediaBox yields no
    usable extent. (GH#1653, GH#1654)

Fixed

  • (ocr): an isolated blank quantity cell in an invoice table can be recovered from its own
    pixels.
    A table whose QTY column has strong integer support — at least two recognized
    whole-number quantities — but exactly one blank cell now gets a single targeted retry that
    re-reads only that cell's bounded pixel region in single-line segmentation mode. The retry never
    derives a quantity from a price or total column, and any table outside this narrow shape takes
    the existing fast path unchanged. A header split across two words (UNIT/PRICE,
    LINE/TOTAL) is also reassembled into one header cell when the fragment sits nearer a
    data-supported column than any standalone column of its own. This OCR quality change ships
    without a corpus-wide A/B measurement.
  • (reranker): a cancelled rerank_async no longer releases its concurrency permit while its
    inference is still running.
    The permit was held by the awaiting future rather than by the
    spawn_blocking task, and a blocking task cannot be cancelled — dropping its handle merely
    detaches it. A caller that timed out or dropped therefore returned the permit immediately while the
    model load and inference it was bounding continued, so repeated abandoned calls could exceed the
    configured reranker concurrency and keep several models resident after every caller had returned.
    (GH#1641)
  • (embeddings, transcription): the same cancelled-permit leak fixed above for the reranker is now
    fixed for embed_texts_async and audio/video transcription.
    Both bound a spawn_blocking
    inference call with a concurrency permit held by the awaiting future rather than by the blocking
    task, so a caller that dropped its future released the permit while the ONNX embedding batch or
    Whisper inference it was bounding kept running. Transcription is timeout-wrapped by default
    (transcription.timeout_ms), so an expiry there was a live, designed-in path to the same
    concurrency breach, not a hypothetical one. (GH#1641)
  • (pdf): a wrapped numbered heading set in a heading-size font no longer loses its second line.
    follows_section's two continuation exemptions only recognized a wrap via a hanging indent or via
    matching right edges, both of which a no-indent heading's short last line fails by construction. A
    third exemption, heading_continuation_at_margin, now recognizes a wrap by measuring the heading's
    own line against the width of the body column beneath it, the same signal the paragraph-merge pass
    already uses for a body-sized heading. Independently, the paragraph-gap detector could still cut a
    continuation these exemptions accepted, because a heading's own line pitch legitimately exceeds the
    body's and was read as a blank-line break; the gap check is now suppressed at exactly the boundary a
    continuation exemption already accepted. (GH#1650)
  • (packaging): the Helm chart is published again. The chart publish was gated such that a failed
    Docker build leg skipped it entirely, while the container images themselves published and the
    release still reported success. No chart was published for 1.1.4, 1.1.5, 1.2.0, 1.2.1 or 1.2.2, so
    helm install --version <current> could not resolve. The chart for 1.2.2 has been published
    retroactively, and a release now fails loudly if its chart is missing rather than shipping without
    one.
  • (node): a JavaScript plugin registered through the Node binding now actually runs. Every
    generated Node trait bridge -- OcrBackend, PostProcessor, Validator, Renderer,
    TokenizerBackend and the rest -- was structurally unable to call back into JavaScript. It
    invoked the JS callable directly from whatever Tokio worker thread the pipeline happened to be
    on, with no N-API handle scope; it never awaited a returned promise, so an async method's
    result was the string "[object Promise]"; and it coerced whatever came back to a string before
    parsing it as JSON, which fails for a plain object as well. The published .d.ts had always
    declared the working contract (processImage(imageBytes: Uint8Array): Promise<ExtractedDocument>),
    so no plugin could have been written against the shipped behaviour. Bridges now hold a
    ThreadsafeFunction created on the JS thread, await the reply through call_async_catch (a JS
    throw surfaces as a plugin error instead of aborting the process), and decode the value without
    the string round-trip. A handle reference that would otherwise be released off its owning JS
    thread is leaked rather than freed, since freeing it there corrupts the V8 heap. (GH#1636)

Changed

  • (pdf): a page whose MediaBox is not rooted at the origin no longer reports the wrong page
    size.
    The native engine's per-page text result took its width and height from the MediaBox
    upper-right corner instead of its extent, so a page with MediaBox [10 -100 622 692] reported
    622x692 rather than the true 612x792. That figure feeds hierarchy reading-order computation, so
    a non-zero MediaBox origin could mis-order content, not merely mis-scale it. Pages rooted at
    (0, 0) — the overwhelming majority — are unaffected, since the two expressions agree there.
    (GH#1653)

  • Upgraded the crawlberg dependency to 1.7.0 (from 1.6.4). xberg re-exports crawlberg's
    crawl types -- CrawlConfig, MapResult, SitemapUrl -- so the pin is part of this crate's
    public surface. 1.7.0 makes no Rust API changes; its breaking changes are confined to
    crawlberg's own Java and Swift bindings, which xberg does not consume.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.2.3/xberg-zig-v1.2.3.tar.gz\",\n        .hash = \"xberg-1.2.3-iV1GrgLORRm2zXRsxaZHWV8cdrQNfPRQsQCDMrPmDJrM\",\n    },\n},\n```\n

v1.2.2

Choose a tag to compare

@Goldziher Goldziher released this 15 Sep 17:55

Fixed

  • (pdf): a two-column page is no longer split down the middle of a column because one line
    crosses its gutter.
    The corridor search treated whitespace as "no span's bbox", so a single
    centred footer, caption, or heading set across both columns closed the page's real gutter for
    every line on the page. The per-line median then placed the split inside the left column, every
    line it crossed became a band boundary, and the reordered text interleaved the two columns
    mid-word — producing spliced tokens such as activamodel and demandviating. A split that six or
    more lines run through now also considers corridors at most one line crosses, still bounded by the
    same distance cap and still refused when either side does not read as a column. Measured over 490
    corpus PDFs: 7 documents change, all of them repairing cross-column splices, and none loses
    vocabulary. Reported and fixed by Data Polder. (GH#1619)
  • (pdf): a wrapped numbered heading now closes above the run-in beneath it. The GH#1634 fix's
    closing term also required a right-edge test that only applies to the un-indented case. A hanging
    indent heading's wrap and a run-in sub-heading below it are both short lines by definition, so
    their right edges land within tolerance of each other by coincidence, which overrode the correct
    left-edge answer and held the heading open across the run-in and the body under it. (GH#1637)
  • (ppt): a legacy .ppt slide's title is read from the file rather than guessed. The title came
    from the first line of the slide's merged text, gated on a second line existing and the first being
    at most 80 characters — so a title-only slide had no title, a two-line title was truncated to its
    first line, and a longer title was dropped. The outline collection's own TextHeaderAtom states
    which text is the title; that is now read, with the first-line rule kept only for decks that state
    no title. (GH#1635)
  • (ppt): the live save's slides are read, not the oldest save's. The persist chain resolved the
    current revision correctly and then scanned the stream for the first SlideListWithText, which on
    a deck saved more than once belongs to the earliest save — so edited titles and outline text
    reverted to a superseded revision. (GH#1639)
  • (ppt): speaker notes attach to the slide that owns them, and the notes master is no longer one.
    The notes master's placeholder text ("Click to edit Master text styles…") was emitted as the first
    slide's speaker note, and the remaining notes were zipped to slides by position, so every note
    after a slide without one landed on the wrong slide. Notes are now keyed by the NotesAtom's own
    slideIdRef. (GH#1640)
  • (release): the xberg-cli archives are attached again. v1.2.0 published 12 assets and
    v1.2.1 published 54, both with none of the nine xberg-cli-* archives, so mise, cargo binstall
    and every direct download had nothing to fetch. One matrix leg (the aarch64-musl link) failed, and
    the upload job required all three CLI build jobs to succeed, so a single leg zeroed the whole asset
    class — as a skipped job, which reports no failure and left both releases looking green. The
    upload now attaches whatever built, and a following step fails the release when any of the nine is
    missing. (GH#1638)
  • (pdf render): s (close-and-stroke) is no longer dropped, and no longer floods the next fill.
    The object-level content parser had arms for every path-painting operator in ISO 32000-1 Table 60
    except s, which fell through to an operator every consumer ignores. Two things followed: the
    stroke was never painted, and — because only a painting arm resets the path builder — the
    abandoned geometry stayed in the builder and was painted by the next fill. A stroked frame
    followed by an ordinary white label fill therefore flooded the frame's whole interior, wiping
    anything already drawn inside it. Text extraction was unaffected: the byte-level fast path always
    decomposed s correctly. (GH#1633)
  • (pdf render): a glyph is no longer discarded because its inferred Unicode is whitespace.
    The byte-indexed rasterizer decided whether to paint by asking what character a code represents
    rather than whether there was an outline to draw. For a simple TrueType font with a byte-indexed
    cmap and no /Encoding that character is the glyph id reverse-mapped through the font's own
    (1, 0) subtable and read as ASCII or Mac Roman — private glyph ordering decoded as an encoding
    it never expressed. Six byte values were affected (0x090x0D, 0x20, and 0xCA → U+00A0);
    re-indexed subsets numbering their glyphs from 0x01 upward walked through a whole run unpainted
    while the advance still applied, leaving gaps that read as spaces. (GH#1632)
  • (pdf): a hanging-number heading no longer swallows body text indented to its title's edge.
    A line was treated as a numbered heading's continuation whenever the heading had a hanging indent
    and the line began at the title's left edge — which is equally the geometry of any layout that
    indents the whole clause, as contracts, tenders and many installation manuals are set. Wrapping
    now also requires that the preceding line actually ran out of room. Separately, a heading that had
    absorbed a genuine wrap could never be closed, so the sub-heading and entire body below were
    pulled in after it. (GH#1634)
  • (cli): the Linux musl CLI binaries are published again.
    Since 1.2.0 the aarch64-unknown-linux-musl CLI build has been killed mid-link, and because
    the upload job requires every musl leg to succeed, no CLI assets were attached to the 1.2.0
    or 1.2.1 releases. The cause was the switch to fat LTO: over the all feature set the final
    whole-program link exceeded what the arm64 runner could complete. That build now uses a
    release-musl profile with thin LTO; every other target keeps fat LTO.
  • (php): setting outputFormat no longer throws. Every call that passed an output format —
    ExtractionConfig::from_json(json_encode(["outputFormat" => "markdown"])), and any
    extract/extractBatch carrying a per-input config — raised
    invalid type: string "markdown", expected struct OutputFormat. The generated Xberg\OutputFormat
    class carried a derived serde implementation that read and wrote its own storage shape,
    {"type":"markdown"}, while the core enum is externally tagged and writes a bare string, so no
    real value ever deserialized. EntityCategory and PiiCategory carried the same defect.
    Introduced in 1.2.0 alongside the flat-class change and shipped in three releases.
  • (wasm): metadata.format fields are readable again. metadata.format.title returned '' and
    metadata.format.sheetCount returned NaN for every document, because the value reached
    JavaScript as a Map rather than the plain object index.d.ts declares — serde-wasm-bindgen
    renders every serde map that way, and serde's internally tagged enum representation is a map.
    Property access on a Map is undefined. Introduced in 1.2.0 with the FormatMetadata flatten.

Changed

  • extraction::ppt::PptSlideText carries the slide's stated title and notes. The struct gains
    title: Option<String> and notes: Option<String>, so Rust callers constructing it with an
    exhaustive struct literal must add the two fields. The type is not exposed through any binding.
  • Breaking (PHP binding): OutputFormat, EntityCategory and PiiCategory now serialize to the
    wire their Rust counterparts emit — "markdown" for a fieldless variant and {"custom":"latex"}
    for one carrying a label — instead of the flat {"type":"markdown"} object. from_json still
    accepts the object form, so existing input keeps working; output moves.
  • Breaking (WebAssembly binding): metadata.format — and any other field carrying a flattened,
    camelCased enum — is now a plain object rather than a Map. This is the shape index.d.ts has
    always declared; code that worked around the discrepancy with .get() must switch to property
    access. Fields carrying an enum that is not flattened, such as chunking.sizing, keep the
    serde-wasm-bindgen bridge and are unchanged.
  • Breaking (C#, Dart, Go, Java, Kotlin, Python, Swift bindings): an enum variant whose name ends
    in a multi-letter acronym followed by a single lowercase letter is spelled correctly now.
    StructuredDataType.RdFa becomes Rdfa (C#, Dart, Go, Swift), Kotlin's R_D_FA becomes RDFA
    and PADDLE_O_C_R becomes PADDLE_OCR, and Python's stub and runtime agree on RDFA. Wire
    values are unchanged — only the identifiers move. Names without that shape (IOError,
    HTMLParser, JSONLD) are unaffected.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.2.2/xberg-zig-v1.2.2.tar.gz\",\n        .hash = \"xberg-1.2.2-iV1GrqlPQRkvcAuD7gyedM9miKz1RpUv6XLEOMIDtJ2f\",\n    },\n},\n```\n

v1.2.1

Choose a tag to compare

@Goldziher Goldziher released this 14 Sep 10:15
v1.2.1
7775e25

Fixed

  • (swift): externally tagged enums now decode the wire the core types actually emit. Regenerated
    on alef 0.87.1. EntityCategory, PiiCategory and ConfidenceSemantics relied on Swift's
    synthesized Codable, which keys every variant ({"person":{}}), while serde writes a bare
    string for a fieldless variant and a single-keyed object whose value is the payload
    ({"custom":"foo"}). Every bridge-constructed value carrying one of these threw
    DecodingError.typeMismatchEntity and PiiEntity decode their category into a
    non-optional field, so any result containing an entity failed. OutputFormat is handled
    separately: its Custom(String) variant carries #[serde(untagged)], so it round-trips as a
    bare string rather than a keyed object.

Changed

  • Dependencies upgraded across the workspace, including crawlberg 1.6.1 → 1.6.3 and
    tree-sitter-language-pack 1.19.0 → 1.19.1. skrifa stays pinned at 0.46: 0.47 moves to
    read-fonts 0.44 while harfrust 0.13.3 is still on 0.43, and the PDF text rasterizer passes a
    harfrust FontRef to skrifa's OutlineFace.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.2.1/xberg-zig-v1.2.1.tar.gz\",\n        .hash = \"xberg-1.2.1-iV1Grpk9ORnRnjY-6afDgG-gPe1D_9elMrRtimTw19ln\",\n    },\n},\n```\n

v1.2.0

Choose a tag to compare

@Goldziher Goldziher released this 14 Sep 04:18
v1.2.0
044bfa0

Added

  • ServerConfig gained job_timeout_secs (default 600 seconds, override via
    XBERG_JOB_TIMEOUT_SECS or server.job_timeout_secs) as the configurable fallback timeout
    for POST /extract-async jobs whose request does not pin down extraction_timeout_secs. A
    per-request extraction_timeout_secs still always overrides it, and an explicit
    extraction_timeout_secs: null still falls back to this server cap rather than running
    unbounded. Previously this fallback was a hardcoded 300 seconds, inconsistent with the 600
    second default used everywhere else. See the Changed section for the source-compatibility impact.

  • A long-running process can now release the embedding and reranker models it no longer
    uses. embeddings::evict_model and reranking::evict_model (and the same functions in
    sparse_embeddings and late_interaction) drop one model, clear_engine_cache drops
    every model in a cache, and xberg::clear_engine_caches drops all of them.
    set_engine_cache_limit bounds the number of resident engines in a cache and drops the
    least recently used one first. The default stays unbounded, so existing callers see no
    change (GH#1626).

Fixed

  • OutputFormat.custom in the Python binding always returned None, even when the value genuinely
    was a custom format. The accessor derived a discriminator from the enum's JSON, which works for
    every tagged representation but not for this variant, whose payload is untagged and so carries no
    discriminator to find. It now matches on the variant directly and returns the label.

  • The Python type stub declared a type: str attribute on 23 enum classes that have no such
    attribute at runtime. The stub emitted it for every data enum, while the runtime only exposes it
    for enums carrying an explicit serde tag -- so for externally tagged enums (EntityCategory,
    PiiCategory, OutputFormat) a type checker accepted category.type, which raised
    AttributeError on use. Stub-only change; no runtime behaviour moved.

  • The Ruby binding's FormatMetadata.from_hash and DiffLine.from_hash read a _0 key that no
    longer exists on the wire, so every variant deserialized with a nil payload. All 24 call
    sites are corrected: the 21 FormatMetadata variants now build their payload from the flattened
    hash, and the 3 DiffLine variants read the text key the enum's #[serde(content = "text")]
    actually emits (#1594).

  • The Python type stub declared format-metadata payloads as _0 (for example
    _0: ExcelMetadata), a key present neither on the wire nor on the runtime object -- the runtime
    per-variant getters such as .excel were always correct. The stub now names the real fields, so
    type checkers stop reporting valid code as an error. Runtime behaviour is unchanged
    (#1594).

  • The Elixir Xberg.FormatMetadata typespec documented a metadata: payload key that matched
    neither the NIF struct nor the serialized wire. It now matches the struct the NIF actually
    returns (#1594).

  • A Type0 (composite) font's content-stream character codes are now translated to CIDs before glyph
    widths and vertical metrics are looked up, instead of being used as if they already were CIDs.
    The two coincide only for Identity-H/Identity-V, which is presumably why this went unnoticed.
    A PDF using a non-Identity predefined CMap (UniCNS-UCS2-H, UniJIS-UCS2-H, UniGB-UCS2-H,
    UniKS-UCS2-H, or their -V/UTF16 counterparts) or an embedded /Encoding CMap stream got a
    wrong width for nearly every glyph -- some over-advancing by up to 4x through the /DW fallback,
    others under-advancing -- which rendered as stretched or overlapping text and, in extracted text,
    could split one sentence into a spurious extra paragraph. CIDs now resolve from an embedded
    /Encoding CMap stream's own begincidrange/begincidchar data when present (including a
    variable-width codespace), else from the font's /CIDSystemInfo character collection for the
    four Unicode-keyed predefined families above, else via Identity-H/Identity-V as before.
    Measured over the 230-document local PDF corpus: 227 byte-identical -- the expected result, since
    most PDFs use Identity-H -- and 2 changed, both merging text that a stale glyph position had
    fragmented. Legacy multi-byte predefined CMaps this crate carries no code-to-CID table for
    (90ms-RKSJ-H, GBK-EUC-H, B5-H, the UTF8 family, UniJIS-UCS2-HW-*, UniJISPro-*, and
    others) are unchanged -- still wrong, not newly broken -- and now log once per font instead of
    failing silently (GH#1631).

  • A reconstructed PDF table cell now reads left to right instead of in the order its words happened
    to arrive. The reported symptom was a sub/superscript printing after the rest of the cell --
    eta_S % came out as eta % S, Q_HE GJ as Q GJ HE -- because a script is drawn as its own
    content-stream segment a fraction of a point below the line it annotates, so every reading-order
    sort upstream placed it after the whole line. The same defect also transposed values between
    columns when two columns were merged into one cell: on a balance sheet whose header reads
    2017 2016, the row beneath it emitted the 2016 figure first, silently attributing each year's
    number to the other year. A cell's words are now grouped into visual lines and ordered left to
    right within each line. Grouping first is load-bearing -- ordering by horizontal position alone
    interleaves the two halves of a wrapped cell. Each word's own whitespace is also collapsed, so a
    segment carrying a trailing space no longer stacks it on the separator. Table cells recovered by
    OCR go through the same ordering (GH#1628).

  • Image OCR now honours a PNG's embedded pHYs pixel density instead of assuming 72 DPI. A genuine
    300-DPI PNG submitted with target_dpi = 300 was resized anyway, because the extractor decoded,
    resized and re-encoded the image -- discarding the density chunk -- before the OCR backend, and
    therefore before the existing ocr.backend_options["source_dpi"] override, ever saw it. Embedded
    density is now resolved at the extractor boundary and at the backend from one shared
    implementation, with the explicit override still taking precedence over it. An image carrying no
    density metadata still defaults to 72 DPI and still resizes (GH#1630).

  • A body paragraph is no longer deleted for repeating text that appears elsewhere on the same page.
    The second strip_repeating_text pass keyed on lowercased paragraph text with no check that a
    table was involved, so a sentence matching an earlier title -- differing only in case, with no
    table on the page at all -- was silently removed. The pass now runs only on pages that have a
    detected table, removes a paragraph only when that table's own cells carry the same text, and
    compares case-sensitively. Measured over 230 PDFs: 44 documents changed, 3472 words recovered and
    32 lost, both loss cases inspected and benign (one is a restructure whose total content grew, the
    other two mojibake tokens) (GH#1623).

  • OCR text is no longer discarded when a scanned page region is detected as a table but its cell
    grid cannot be recognised. recognize_single_table returned nothing whenever TATR failed,
    produced no rows or columns, or the grid failed validation, which threw away every OCR element
    that had been assigned to that region. A region that cannot be recognised as a table now falls
    back to emitting its text in reading order, and only when that text is not already carried by
    one of the page's paragraphs, so nothing is duplicated. Together with the restructuring-heuristic
    retention guard below, recognised OCR text is no longer silently lost on the layout path
    (GH#1622).

  • extraction_confidence no longer reports a failed structured extraction as fully
    schema-valid. The pipeline passed SchemaCompliance::AllValid unconditionally, which is 40% of
    the combined score under the default weights, so a run whose LLM call failed -- or that was
    built without the liter-llm feature, or ran on wasm -- scored exactly as high as one that
    validated. A requested structured_extraction that leaves no structured_output now scores
    AllInvalid. Extractions with no structured_extraction configured are unaffected and keep
    their previous score; ConfidenceSignals is unchanged in shape, so no serialized form moves
    (GH#1624).

  • detect_mime_type_from_bytes no longer refuses text that is not valid UTF-8. A byte buffer with
    no filename or declared type -- a Windows-1252 or ISO-8859-1 CSV export, say -- returned
    UnsupportedFormat even though the extractors that would receive it decode legacy encodings
    through encoding_rs. Such content is now reported as text/plain, the same answer the UTF-8
    path already gave for the same document, so the two encodings of one file behave alike. Content
    holding a NUL byte, or with too few printable bytes to read as prose, is still rejected
    (GH#1625).

  • The Go binding no longer discards the message of every error the native layer reports. Each
    known error code was mapped to a typed sentinel (ErrTimeout, ErrParsing, ErrOcr, and ~20
    more) and returned before the message was ever read, so the detail the native layer had
    already produced -- observed durations, limits, plugin names, counts -- was dropped for all of
    them; only unrecognised codes kept their text. A timeout surfaced as the sentinel's own
    placeholder-stripped text, extraction timed out after ms (limit: ms), which reads as a
    formatting bug but is the whole message the binding ever had, and left callers unable to tell
    which timeout had fired. The message is now read first and returned alongside the sentinel, so
    errors.Is(err, xberg.ErrTimeout) sti...

Read more

v1.1.5

Choose a tag to compare

@Goldziher Goldziher released this 10 Sep 04:29

Fixed

  • The Java binding compiles again. A method returning Option<Vec<u8>>Registry.sampleBytes
    is the only one today — was generated declaring Optional<byte[]> while returning a bare
    byte[], which javac rejects. 1.1.4 therefore published no Java artifact at all, and the
    spring-ai integration was blocked waiting on it. Fixed upstream in alef 0.85.12; this release
    regenerates on it.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.1.5/xberg-zig-v1.1.5.tar.gz\",\n        .hash = \"xberg-1.1.5-iV1GrvucGBnYUxPwjVLLE8FduJyau_QIJfXxreCBEDBT\",\n    },\n},\n```\n

v1.1.4

Choose a tag to compare

@Goldziher Goldziher released this 09 Sep 18:47

Fixed

  • PDF reading order no longer tears a subscript off the symbol it names. Spans were ordered by the
    top of their bounding box, but a subscript is drawn 35-40% smaller than its base, so its top sits
    several points lower even though its baseline is a fraction of a point away. An unrelated span
    from the next column could sort between a base run and its own subscript, and the symbol the
    subscript names no longer existed anywhere in the output. Ordering now quantises the baseline
    into row bands before comparing horizontally, which is what every other caller of that comparator
    already did (GH#1600).
  • PDF table detection no longer bridges two separate tables across the graphics-free gap between
    them. A cell was built from intersection points alone, so a section heading printed in that gap
    was absorbed into one of the tables as a single-cell row. A candidate cell now also requires a
    drawn vertical rule spanning its own Y-range on both sides. The span tolerance is load-bearing:
    at the tighter X-axis value, rows of a table whose rules are inset by a few points are dropped
    (GH#1601).
  • PDF two-column detection no longer loses the page's split to a hanging-number indent. When any
    span straddled a correctly detected gutter, the split was replaced outright by the midpoint of
    the widest whole-page whitespace corridor — on a hanging-number layout, the indent between the
    numbers and the text. The reorder then hoisted every clause number out of its clause. A
    relocation is now rejected when it would move the split more than a quarter of the page width,
    which leaves every legitimate corridor move in the corpus intact (GH#1603).
  • PDF paragraph grouping no longer splits a numbered heading that wraps onto a shorter second line.
    The wrap exemption compared the two lines' right edges, but a heading fills its column on its
    FIRST line and the continuation is whatever is left over, so the metric was anti-correlated with
    the answer. The pair is now also exempt when the continuation opens lowercase AND the heading
    line reaches within a tolerance of the width of what would be merged onto it — the "fills its
    column" half the original rule stated but never measured. The lowercase test alone is not
    sufficient: body prose beginning lowercase under a complete numbered heading has the same
    signature (GH#1605).
  • PDF paragraph grouping now recognises a numbered heading whose line arrives as more than one text
    span. The break terms tested the predicate against a single span, so a heading set with a hanging
    section number — 3.1.7 in one span, its title in the next, on one baseline — never looked like a
    numbered heading and was left to the ordinary paragraph-gap rule. That rule needs a gap wider than
    ordinary line pitch, so every such heading whose body starts on the next line was welded into it.
    The line's spans are now re-joined before the predicate runs, which is what the continuation-merge
    pass already did (GH#1609).
  • PDF paragraph grouping now recognises a heading whose number is not its first token — ARTIKEL 1.,
    Chapter 1, Appendix 1, Annex III, Exhibit A. The numbered-heading predicate is the only
    boundary signal available when a heading shares font, size, weight and leading with its
    neighbour, so a heading it could not see was welded onto the line above it, and a run of such
    headings collapsed into a single element. Recognition is by shape, not by a keyword list: one
    capitalised word standing in front of an enumerator. Prose that opens the same way — Artikel 12 van de wet is van toepassing. — stays prose, because behind a keyword the text after the
    enumerator must still be capitalised (GH#1608).
  • PDF heading detection no longer skips a numbered heading that is only two words long. Promotion
    of a bold, body-size line to a heading required more than two words — a floor that keeps short
    bold fragments out — and a numbered section title such as 3. PRIJZEN or 1. INTRODUCTION falls
    below it. Those lines stayed plain bold paragraphs, and a run of them was then coalesced into a
    single bold line in the rendered output, while the element stream still reported them separately.
    A numbered section heading is now exempt from the word-count floor; everything else still has to
    clear it (GH#1611).
  • OCR no longer adopts a markdown table rebuild that loses content. The rebuilt page replaced the
    original whenever it was merely non-empty, so a rebuild that dropped text still won. The rebuild
    is now rejected, with a warning naming both word counts, when it retains fewer words than the
    content it would replace (GH#1599).
  • PaddleOCR's default model_tier of mobile now resolves to the pp-ocrv6 small detection model
    (9.9 MB) rather than medium (62 MB). A tier named mobile silently loading the largest
    available model made a 21-page document take over ten minutes. small and medium share the
    same 18,708-character dictionary, so recognition coverage is unchanged. The documented model
    sizes were also wrong and have been corrected (GH#1602).
  • The PHP extension now loads on Debian 12 and other distributions built against GCC 12. The Linux
    publish runners ship GCC 13+, and the extension picked up a GLIBCXX_3.4.31 symbol from their
    libstdc++ while Debian 12 provides at most GLIBCXX_3.4.30. libstdc++ is now linked statically;
    the highest glibc requirement was already below Debian 12's (GH#1606).

Zig

Add to your build.zig.zon:

.dependencies = .{
    .xberg-zig = .{\n        .url = \"https://github.com/xberg-io/xberg/releases/download/v1.1.4/xberg-zig-v1.1.4.tar.gz\",\n        .hash = \"xberg-1.1.4-iV1GrisMGRnWze1J7F5S-47W47m4kvC81K2vj_OMtHA-\",\n    },\n},\n```\n