Repository navigation
feat: discover duplicate edges with uncapped IVF radius queries - #10304
jackye1995 wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
Automatic probing now considers at most two centroids per target segment by default, so threshold matches can be omitted even with IVF_FLAT. Raising max_nprobes broadens discovery; quantization retains its separate recall trade-off.
The partition-level approach remains reasonable when partitions fit worker memory. Whole-partition loading and per-query candidate buffering mean the smaller source and verification batches still do not establish a memory budget.
Tests cover L2/IVF_FLAT, including nested projections and the probe-budget behavior. I could not run them because the required Rust 1.98.1 is unavailable and cannot be installed into this environment’s read-only rustup directory. Quantized, other-metric, and multi-segment behavior remains unverified end to end.
Add an alternative to comparing only vectors stored in the same IVF partition: use each live source vector as a query against IVF cells in every active index segment, then emit threshold-matching edges.
Execution flow
find_similar_pairs_in_partitiondiscovers edges for one source scope. It searches all target segments, so matches may cross the source partition and segment boundary. Radius scans return every qualifying candidate within the scanned cells; no top-k truncation is applied. Self edges are excluded. Discovery can be asymmetric, so matches are retained regardless of which endpoint found the other; repeated canonical edges may be emitted and can be merged by the caller.Probe selection and distance semantics
nprobesselects a fixed number of nearest cells.max_nprobeschanges the automatic candidate budget; counts must be positive and fixed probes cannot exceed an explicit maximum. Budgets apply independently to each target segment.Requires one current-format logical vector index covering the pinned dataset, with stale indexed values rejected. Source/target range blocking is a separate change in #10305.
Coverage includes centroid-boundary matches, more than 100 radius neighbors, nested vector columns and automatic/fixed probe budgets.