Skip to content

feat(pyramid): add extra_info support for build, search, and serialization - #2994

Open
LHT129 wants to merge 1 commit into
antgroup:mainfrom
LHT129:deepseek-harness/pyramid-extra-info
Open

LHT129 wants to merge 1 commit into
antgroup:mainfrom
LHT129:deepseek-harness/pyramid-extra-info

Conversation

@LHT129

@LHT129 LHT129 commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds full extra_info support to the Pyramid index, aligning it with HGraph.

Changes

  • Build/Add: Write extra_info in encode_add_batch() and build_by_odescent()
  • Search: Return extra_info in search results via search_impl()
  • Serialization: Add extra_info block to Serialize(), Deserialize(), streaming serialize_streaming_body() and read_streaming_body()
  • Resize: Include extra_infos_ resize in resize()
  • Memory: Add extra_infos to GetMemoryUsageDetail()
  • Feature flags: Register SUPPORT_GET_EXTRA_INFO_BY_ID, SUPPORT_KNN_SEARCH_WITH_EX_FILTER, and SUPPORT_UPDATE_EXTRA_INFO_CONCURRENT when extra_info is configured
  • Parameter mapping: Accept extra_info key in CheckAndMappingExternalParam() and add default extra_info config in build_default_pyramid_param()
  • Analyzer: Relax AnalyzeIndexBySearch() to allow id-filter and bitset-filter while keeping the attribute/iterator filter guard
  • Tests: Add pyramid extra_info tests covering build, search, get-by-id, file serialization roundtrip, and binary set serialization roundtrip

Copilot AI lite review requested due to automatic review settings September 22, 2026 10:15
@vsag-bot

vsag-bot commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

@pull-request-size pull-request-size Bot added the size/L 100-499 changed lines label Sep 22, 2026
@mergify mergify Bot added module/index Index algorithms and implementations 索引算法与实现 area/testing Tests, fixtures, and test infrastructure 测试、夹具与测试基础设施 labels Sep 22, 2026
@mergify

mergify Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Require kind label

  • label~=^kind/

🟢 Require version label

  • label~=^version/

Comment thread src/algorithm/pyramid/pyramid.cpp
Comment thread src/algorithm/pyramid/pyramid.cpp
Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@LHT129 LHT129 added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/1.0 1. LazyHGraph & SIMQ 2. Unified search API 3. Streaming serialization 1. 新索引 2. 统一检索接口 3. 流式序列化 labels Sep 22, 2026
@vsag-bot vsag-bot removed the version/1.0 1. LazyHGraph & SIMQ 2. Unified search API 3. Streaming serialization 1. 新索引 2. 统一检索接口 3. 流式序列化 label Sep 22, 2026
@LHT129 LHT129 added kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 version/1.0 1. LazyHGraph & SIMQ 2. Unified search API 3. Streaming serialization 1. 新索引 2. 统一检索接口 3. 流式序列化 and removed kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/1.1 labels Sep 22, 2026
@vsag-bot vsag-bot added version/1.1 and removed version/1.0 1. LazyHGraph & SIMQ 2. Unified search API 3. Streaming serialization 1. 新索引 2. 统一检索接口 3. 流式序列化 labels Sep 22, 2026
Copilot AI review requested due to automatic review settings September 23, 2026 02:43
@LHT129
LHT129 force-pushed the deepseek-harness/pyramid-extra-info branch from 89af147 to e5def65 Compare September 23, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/algorithm/pyramid/pyramid.cpp

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR adds comprehensive extra_info support to the Pyramid index, aligning it with the existing HGraph implementation. The changes are well-structured and cover all the necessary paths: build, add, search, serialization (both legacy and streaming), deserialization, resize, memory tracking, and feature flags.

What was reviewed

  • src/algorithm/pyramid/pyramid.cpp (+75/-6): Core extra_info integration
  • tests/test_pyramid.cpp (+78): Tests for build, search, get-by-id, and serialization roundtrips

Observations

The implementation follows the same patterns used by HGraph for extra_info support, which is good for consistency. The feature flags registered in InitFeatures() match the capabilities being added. The serialization changes correctly handle both the legacy Serialize/Deserialize path and the streaming serialize_streaming_body/read_streaming_body path.

Three inline comments have been left on specific areas that could benefit from additional guards (see inline discussions).

@LHT129
LHT129 force-pushed the deepseek-harness/pyramid-extra-info branch from e5def65 to f53adac Compare September 23, 2026 04:15
Copilot AI review requested due to automatic review settings September 23, 2026 04:15
Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread tests/test_pyramid.cpp
…ation

Add extra_info write path in encode_add_batch() and build_by_odescent(),
extra_info extraction in search_impl(), extra_info serialization in
Serialize()/Deserialize(), streaming serialize/deserialize EXTRA_INFO block,
resize() support, and GetMemoryUsageDetail() reporting.

Register SUPPORT_GET_EXTRA_INFO_BY_ID, SUPPORT_KNN_SEARCH_WITH_EX_FILTER,
and SUPPORT_UPDATE_EXTRA_INFO_CONCURRENT feature flags when extra_info is
enabled.

Add extra_info parameter mapping in CheckAndMappingExternalParam() and
default extra_info config in build_default_pyramid_param().

Relax AnalyzeIndexBySearch filter restriction to allow id-filter and
bitset-filter while keeping the attribute/iterator filter guard.

Signed-off-by: tianlan.lht <tianlan.lht@antgroup.com>
Assisted-by: DeepSeekV4Pro:deepseek-v4-pro
@LHT129
LHT129 force-pushed the deepseek-harness/pyramid-extra-info branch from f53adac to b47c287 Compare September 23, 2026 06:37
Copilot AI review requested due to automatic review settings September 23, 2026 06:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

CHECK_ARGUMENT(extra_infos_->max_capacity_ >= static_cast<uint64_t>(data_num),
"extra infos capacity is smaller than the data count");
for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) {
extra_infos_->InsertExtraInfo(extra_infos + inner_id * extra_info_size_, inner_id);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] Incorrect extra_infos indexing in build_by_batch_graph non-optimized path. The loop uses inner_id to index into the input extra_infos array, but when data_num != input_count (e.g., PIPNN deduplication path at line 622-628), input_indices is not a contiguous 0..N-1 sequence. The other data components in the same code block correctly use input_indices[inner_id] for indexing (see insert_codes lambda at line 672-675), but extra_info does not.

Current code:

for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) {
    extra_infos_->InsertExtraInfo(extra_infos + inner_id * extra_info_size_, inner_id);
}

Suggested fix:

for (InnerIdType inner_id = 0; inner_id < data_num; ++inner_id) {
    extra_infos_->InsertExtraInfo(extra_infos + input_indices[inner_id] * extra_info_size_, inner_id);
}

Note: the optimized path through encode_add_batch already uses input_index correctly (line 1892).

insert_codes(raw_vector_);
}
const auto* extra_infos = base->GetExtraInfos();
if (extra_infos_ != nullptr && extra_infos != nullptr) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The storage_preallocated condition in build_by_batch_graph (line 640-643) and prepare_add_batch (line 1817-1821) does not include extra_infos_. While resize() already grows extra_infos_ and encode_add_batch has a capacity guard, the storage_preallocated flag controls whether parallel encoding is used. If extra_infos_ capacity is insufficient but other components are fine, storage_preallocated could be false (or true in prepare_add_batch), leading to a potentially confusing error path instead of a clean preallocation.

Consider adding (extra_infos_ == nullptr or data_num > static_cast<int64_t>(extra_infos_->max_capacity_)) to both storage_preallocated conditions for consistency with the other data components.

}

if (extra_info_size_ > 0 && extra_infos_ != nullptr) {
extra_infos_->Deserialize(buffer_reader);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Missing post-loop validation for loaded_extra_info in read_streaming_body. The EXTRA_INFO block is registered as critical in the streaming manifest (line 1432: StreamSerializationTagCritical(tag)), but after the read loop there is no check to ensure it was actually loaded when extra_info_size_ > 0 && extra_infos_ != nullptr. Other optional-but-critical blocks (precise_codes, raw_vector, paths) all have corresponding post-loop checks (lines 1678-1688), so a stream missing the EXTRA_INFO block would silently succeed without loading extra_info data.

Consider adding a check after the loop:

if (extra_info_size_ > 0 && extra_infos_ != nullptr && !loaded_extra_info) {
    throw VsagException(ErrorType::READ_ERROR,
                        "Pyramid streaming serialization extra_info block is missing");
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Tests, fixtures, and test infrastructure 测试、夹具与测试基础设施 kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 module/index Index algorithms and implementations 索引算法与实现 size/L 100-499 changed lines version/1.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants