Conversation
|
/label status/waiting-for-review |
Merge Protections🟢 All 2 merge protections satisfied — ready to merge. Show 2 satisfied protections🟢 Require kind label
🟢 Require version label
|
89af147 to
e5def65
Compare
LHT129
left a comment
There was a problem hiding this comment.
Review Summary
This PR adds comprehensive extra_info support to the Pyramid index, aligning it with the existing HGraph implementation. The changes are well-structured and cover all the necessary paths: build, add, search, serialization (both legacy and streaming), deserialization, resize, memory tracking, and feature flags.
What was reviewed
src/algorithm/pyramid/pyramid.cpp(+75/-6): Core extra_info integrationtests/test_pyramid.cpp(+78): Tests for build, search, get-by-id, and serialization roundtrips
Observations
The implementation follows the same patterns used by HGraph for extra_info support, which is good for consistency. The feature flags registered in InitFeatures() match the capabilities being added. The serialization changes correctly handle both the legacy Serialize/Deserialize path and the streaming serialize_streaming_body/read_streaming_body path.
Three inline comments have been left on specific areas that could benefit from additional guards (see inline discussions).
e5def65 to
f53adac
Compare
…ation Add extra_info write path in encode_add_batch() and build_by_odescent(), extra_info extraction in search_impl(), extra_info serialization in Serialize()/Deserialize(), streaming serialize/deserialize EXTRA_INFO block, resize() support, and GetMemoryUsageDetail() reporting. Register SUPPORT_GET_EXTRA_INFO_BY_ID, SUPPORT_KNN_SEARCH_WITH_EX_FILTER, and SUPPORT_UPDATE_EXTRA_INFO_CONCURRENT feature flags when extra_info is enabled. Add extra_info parameter mapping in CheckAndMappingExternalParam() and default extra_info config in build_default_pyramid_param(). Relax AnalyzeIndexBySearch filter restriction to allow id-filter and bitset-filter while keeping the attribute/iterator filter guard. Signed-off-by: tianlan.lht <tianlan.lht@antgroup.com> Assisted-by: DeepSeekV4Pro:deepseek-v4-pro
f53adac to
b47c287
Compare
| CHECK_ARGUMENT(extra_infos_->max_capacity_ >= static_cast<uint64_t>(data_num), | ||
| "extra infos capacity is smaller than the data count"); | ||
| for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) { | ||
| extra_infos_->InsertExtraInfo(extra_infos + inner_id * extra_info_size_, inner_id); |
There was a problem hiding this comment.
[critical] Incorrect extra_infos indexing in build_by_batch_graph non-optimized path. The loop uses inner_id to index into the input extra_infos array, but when data_num != input_count (e.g., PIPNN deduplication path at line 622-628), input_indices is not a contiguous 0..N-1 sequence. The other data components in the same code block correctly use input_indices[inner_id] for indexing (see insert_codes lambda at line 672-675), but extra_info does not.
Current code:
for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) {
extra_infos_->InsertExtraInfo(extra_infos + inner_id * extra_info_size_, inner_id);
}Suggested fix:
for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) {
extra_infos_->InsertExtraInfo(extra_infos + input_indices[inner_id] * extra_info_size_, inner_id);
}Note: the optimized path through encode_add_batch already uses input_index correctly (line 1892).
| insert_codes(raw_vector_); | ||
| } | ||
| const auto* extra_infos = base->GetExtraInfos(); | ||
| if (extra_infos_ != nullptr && extra_infos != nullptr) { |
There was a problem hiding this comment.
[suggestion] The storage_preallocated condition in build_by_batch_graph (line 640-643) and prepare_add_batch (line 1817-1821) does not include extra_infos_. While resize() already grows extra_infos_ and encode_add_batch has a capacity guard, the storage_preallocated flag controls whether parallel encoding is used. If extra_infos_ capacity is insufficient but other components are fine, storage_preallocated could be false (or true in prepare_add_batch), leading to a potentially confusing error path instead of a clean preallocation.
Consider adding (extra_infos_ == nullptr or data_num > static_cast<int64_t>(extra_infos_->max_capacity_)) to both storage_preallocated conditions for consistency with the other data components.
| } | ||
|
|
||
| if (extra_info_size_ > 0 && extra_infos_ != nullptr) { | ||
| extra_infos_->Deserialize(buffer_reader); |
There was a problem hiding this comment.
[suggestion] Missing post-loop validation for loaded_extra_info in read_streaming_body. The EXTRA_INFO block is registered as critical in the streaming manifest (line 1432: StreamSerializationTagCritical(tag)), but after the read loop there is no check to ensure it was actually loaded when extra_info_size_ > 0 && extra_infos_ != nullptr. Other optional-but-critical blocks (precise_codes, raw_vector, paths) all have corresponding post-loop checks (lines 1678-1688), so a stream missing the EXTRA_INFO block would silently succeed without loading extra_info data.
Consider adding a check after the loop:
if (extra_info_size_ > 0 && extra_infos_ != nullptr && !loaded_extra_info) {
throw VsagException(ErrorType::READ_ERROR,
"Pyramid streaming serialization extra_info block is missing");
}
Summary
Adds full extra_info support to the Pyramid index, aligning it with HGraph.
Changes