fix(iceberg): merge manifests during COW overwrite - #26992

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Forward-port #26990 to RisingWave main. COW publication should prune obsolete deletion-only manifests and merge small data manifests during the existing overwrite commit, preventing metadata growth while the live file set stays small.

Update the pinned dependencies to iceberg-rust#234, commit 54d7ddc4565254fb284d0199edd0f8d331d0e3f9, and a compaction pin-alignment commit. Both are based on the exact dependencies already used by RisingWave main; no unrelated upstream dependency changes are pulled in. The lockfile changes only these two Git revisions and resolves a single Iceberg source.

Main's Iceberg dependency line no longer contains the old inline merge implementation, so the dependency patch adapts it into one private snapshot-producer helper. It retains the 8 MiB target / 100-manifest first-bin threshold, groups by partition spec, preserves original entry sequences and the current commit's added/deleted entries, and honors explicit property overrides. It adds no extra snapshot or data-file rewrite. No tests or V3-specific handling are added.

Validation:

• 141 existing Iceberg transaction tests pass; the affected test modules are unchanged.
• Iceberg Clippy with warnings denied and workspace formatting pass.
• Compaction RUSTUP_TOOLCHAIN=nightly-2026-03-05 make check passes.
• RW CXXFLAGS='-include cerrno' cargo +nightly-2026-06-21 check -p risingwave_storage --lib --locked passes. The local C++ flag supplies a missing FAISS header and is not part of the source change.
• RW formatting, locked metadata resolution and diff checks pass.
• External in-memory smoke validation: start with 100 manifests and execute three partial overwrites; each yields one manifest with exactly 100 expected live files and the current deletion, preserving original sequences, historical manifest readability and exactly one snapshot per operation. The smoke program is not part of any PR.
• Request Iceberg e2e coverage alongside default PR CI.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests. (No new tests; existing tests and external smoke validation pass.)
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run benchmarks and present the results.
• I have checked which release branches need this fix. (release-3.0 counterpart: fix(iceberg): merge COW overwrite manifests on release-3.0 #26990.)

Documentation

• My PR needs documentation updates. (Internal manifest maintenance; no supported user interface or operational workflow changes.)

Summary by CodeRabbit

• Chores
  • Updated underlying table-format and storage components to newer revisions.
  • Included the latest available improvements and compatibility updates from these components.
  • Added an internal performance benchmark for storage-related operations.
  • No user-facing configuration or feature changes were introduced.