Skip to content

fix(query): paginate result sets larger than the prefetch#8

Open
ysnknr wants to merge 1 commit into
stiang:mainfrom
ysnknr:fix/query-pagination
Open

fix(query): paginate result sets larger than the prefetch#8
ysnknr wants to merge 1 commit into
stiang:mainfrom
ysnknr:fix/query-pagination

Conversation

@ysnknr

@ysnknr ysnknr commented Jun 8, 2026

Copy link
Copy Markdown

Problem

A SELECT that returns more rows than the prefetch size (100) silently stops at the first batch: query() returns 100 rows with has_more_rows = false, so callers treat a truncated result as complete. And calling fetch_more to continue desyncs the connection — the next call dies with I/O error: early eof.

Reproduced against a live Oracle XE 21c:

SELECT object_name FROM all_objects        -- 10,342 rows
  → query() returns 100, has_more_rows = false   (before)

Root causes & fixes

  1. has_more_rows derivation. The first-batch parser (parse_query_response_with_columns) discarded the more-rows signal and hard-coded has_more_rows: false. The correct signal is the end-of-call terminator: the cursor is exhausted iff the response carries ORA-01403 (no_data_found). (Note: a cumulative row_count > 0 is not a valid signal — for a SELECT it is always > 0, which over-fetches past the last row and trips a server MARKER/reset.) parse_error_info_with_rowcount now returns more_rows = error_code != 1403 and the parser propagates it.
  2. FETCH sequence number. FetchMessage::build_request hard-coded the TTC sequence byte to 0; it now carries the connection's next_sequence_number() (the execute path already does this).
  3. FETCH packet framing. FetchMessage always wrote a 2-byte packet length, but on a large-SDU connection the length header is 4 bytes (as the execute path frames it). The mismatched length made the server reject the packet with a MARKER and desync the connection. build_request now takes large_sdu and writes a 4-byte length accordingly.
  4. fetch_more parsing. Parse the fetched batch with the main parse_query_response_with_columns (handles row headers, bit vectors, the terminal message, and sets has_more_rows/cursor_id) instead of the narrower parse_fetch_response, which under-ran on real row data.

Validation

  • Live Oracle XE 21c: query() + while has_more_rows { fetch_more(cursor_id, &cols, 100) } returns all 10,342 rows of all_objects, and a subsequent query on the same connection still works (no early eof). Boundary sizes 5 / 100 / 101 / 250 are all exact, with has_more_rows correct on the first batch.
  • cargo test --lib311 passed, 0 failed.

Known limitation (follow-up)

A single fetch batch that spans more than one TNS packet still truncates (fetch_more reads one packet). Keeping the fetch size so each batch fits one packet (the default prefetch of 100 does this for typical rows) avoids it; full multi-packet accumulation on the fetch path is a separate change.

A SELECT returning more rows than the prefetch (100) silently stopped at the
first batch and could not be continued: query() reported has_more_rows=false,
and calling fetch_more desynced the connection (the next call died with
"early eof"). This makes fetch_more actually work, so query() + a
`while has_more_rows { fetch_more(..) }` loop returns the whole result set.

Four fixes:
- has_more_rows: derive it from the end-of-call terminator (ORA-01403 =
  no_data_found) instead of a hard-coded false / a cumulative row_count that is
  always > 0 for a SELECT (which over-fetches past the last row and trips a
  server MARKER/reset).
- FETCH message: carry the connection's TTC sequence number (was hard-coded 0).
- FETCH packet framing: use the 4-byte length header on large-SDU connections
  (matching the execute path); a 2-byte length made the server reject the
  packet with a MARKER and desync the connection.
- fetch_more: parse the response with the main query-response parser (row
  headers, bit vectors, terminal message) instead of the narrower one.

Validated against a live Oracle XE 21c: a 10,342-row SELECT now returns all
10,342 rows across batches with the connection still usable afterward; boundary
sizes (5/100/101/250) are exact. 311 unit tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant