fix(query): paginate result sets larger than the prefetch#8
Open
ysnknr wants to merge 1 commit into
Open
Conversation
A SELECT returning more rows than the prefetch (100) silently stopped at the
first batch and could not be continued: query() reported has_more_rows=false,
and calling fetch_more desynced the connection (the next call died with
"early eof"). This makes fetch_more actually work, so query() + a
`while has_more_rows { fetch_more(..) }` loop returns the whole result set.
Four fixes:
- has_more_rows: derive it from the end-of-call terminator (ORA-01403 =
no_data_found) instead of a hard-coded false / a cumulative row_count that is
always > 0 for a SELECT (which over-fetches past the last row and trips a
server MARKER/reset).
- FETCH message: carry the connection's TTC sequence number (was hard-coded 0).
- FETCH packet framing: use the 4-byte length header on large-SDU connections
(matching the execute path); a 2-byte length made the server reject the
packet with a MARKER and desync the connection.
- fetch_more: parse the response with the main query-response parser (row
headers, bit vectors, terminal message) instead of the narrower one.
Validated against a live Oracle XE 21c: a 10,342-row SELECT now returns all
10,342 rows across batches with the connection still usable afterward; boundary
sizes (5/100/101/250) are exact. 311 unit tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
SELECTthat returns more rows than the prefetch size (100) silently stops at the first batch:query()returns 100 rows withhas_more_rows = false, so callers treat a truncated result as complete. And callingfetch_moreto continue desyncs the connection — the next call dies withI/O error: early eof.Reproduced against a live Oracle XE 21c:
Root causes & fixes
has_more_rowsderivation. The first-batch parser (parse_query_response_with_columns) discarded the more-rows signal and hard-codedhas_more_rows: false. The correct signal is the end-of-call terminator: the cursor is exhausted iff the response carriesORA-01403(no_data_found). (Note: a cumulativerow_count > 0is not a valid signal — for a SELECT it is always > 0, which over-fetches past the last row and trips a server MARKER/reset.)parse_error_info_with_rowcountnow returnsmore_rows = error_code != 1403and the parser propagates it.FetchMessage::build_requesthard-coded the TTC sequence byte to0; it now carries the connection'snext_sequence_number()(the execute path already does this).FetchMessagealways wrote a 2-byte packet length, but on a large-SDU connection the length header is 4 bytes (as the execute path frames it). The mismatched length made the server reject the packet with a MARKER and desync the connection.build_requestnow takeslarge_sduand writes a 4-byte length accordingly.parse_query_response_with_columns(handles row headers, bit vectors, the terminal message, and setshas_more_rows/cursor_id) instead of the narrowerparse_fetch_response, which under-ran on real row data.Validation
query()+while has_more_rows { fetch_more(cursor_id, &cols, 100) }returns all 10,342 rows ofall_objects, and a subsequent query on the same connection still works (noearly eof). Boundary sizes 5 / 100 / 101 / 250 are all exact, withhas_more_rowscorrect on the first batch.cargo test --lib→ 311 passed, 0 failed.Known limitation (follow-up)
A single fetch batch that spans more than one TNS packet still truncates (
fetch_morereads one packet). Keeping the fetch size so each batch fits one packet (the default prefetch of 100 does this for typical rows) avoids it; full multi-packet accumulation on the fetch path is a separate change.