Tags: infiniflow/infinity
Tags
fix(fulltext): stop BMW scoring iterators on a stale posting block (#… …3496) ### What Fix a server abort in fulltext search with many query terms. Fixes #3495. `match_text` with more than 15 terms could terminate the process from a query thread: ``` TerminateHandler: Unhandled Exception: number of bits is unsupported ``` (or `Decode posting FAILEDF@src/storage/common/int_encoder.cppm:109`), thrown under `BlockMaxWandIterator::Next` -> `TermDocIterator::BM25Score` -> `SkipIndexDecoder::DecodeCurrentTFBuffer`. ### Approach - `BlockMaxWandIterator::Next` only partially sorted `sorted_iterators_` for more than `SORT_SKIP_THRESHOLD` terms. With the pivot in the unsorted tail, iterators in `[0, pivot]` could sit on an earlier doc; BMW called `NextShallow(d)` on them (block cursor moves, nothing decoded) and then `BM25Score()` without a `Next(d)`. The partial sort is removed, the iterators are always fully sorted. - `PostingIterator::SkipTo` now invalidates `current_row_id_` when the block cursor moves, so `GetCurrentTF` / `GetCurrentTTF` / `GetCurrentDocPayload` return 0 instead of decoding from the reader's stale position. Before, the only guard was an `assert` in `MultiPostingDecoder::DecodeCurrentTFBuffer`, compiled out in release builds. - `FragmentTask::OnExecute` catches `std::exception` and fails the query. FastPFor throws `std::logic_error`, which nothing caught before.
Fix: Multiple Memory Indexers Cause BM25 Calculation Error (#3340) Close #3339 ### Summary `ColumnIndexReader` may contain multiple memory indexers (one per segment). However, `FullTextColumnLengthReader::GetDocTermCount()` was only using a single `memory_indexer_` variable, causing `total_df_ < doc_freq_` crash during BM25 calculation. ### Reproduction Steps 1. Set `optimize_interval` to 3600s 2. Run `TestMultipleIndexTypesImport` with snapshot restore enabled 3. The test will crash with `total_df_ < doc_freq_` error ### Expected behavior _No response_ ### Additional information `UnrecoverableError(fmt::format("total_df_ {} is less than doc_freq_ {}", total_df_, doc_freq_));`
Fix PhysicalFilter::Execute (#3342) ### What problem does this PR solve? When table is empty, prev_op_state->data_block_array_.empty() is true in PhysicalFilter, we should return empty result instead of crash. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue)
PreviousNext