Spark 3.5: Add row lineage rule for DELETE to keep plans resolved - #18159

Closes #18131

Problem

A copy-on-write DELETE on a v3 table fails Spark's per-rule plan validation:

[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule GroupBasedRowLevelOperationScanPlanning
in batch Early Filter and Projection Push-Down generated an invalid plan:
The plan becomes unresolved: 'ReplaceData RelationV2[id, data] ...
+- Filter NOT ((id = 101) <=> true)
   +- RelationV2[id, data, _file, _pos, _row_id, _last_updated_sequence_number] ...

SparkCopyOnWriteScan.rowLineageAsDataCols removes the __metadata_col marker from
_row_id and _last_updated_sequence_number so the 3.5 row-level rules do not drop them.
ReplaceData.dataInput filters out metadata attributes, so both lineage columns stay in it,
giving four columns against a target relation that supplies two, and outputResolved fails.
Only the intermediate plan is affected, which is why this surfaces just under plan validation.

Spark 4.x does not override readSchema() at all and relies on the native metadata column
semantics from SPARK-50820, matching the version split in the issue.

Fix

UPDATE and MERGE already append the lineage attributes to the target relation output in
RewriteUpdateTableForRowLineage and RewriteMergeIntoTableForRowLineage. DELETE had no
equivalent, so this adds RewriteDeleteFromTableForRowLineage alongside them. It needs no
assignments because copy-on-write DELETE carries surviving rows over unchanged.

shouldUpdatePlan now also returns false for non-Iceberg tables. Its inner match had no
default branch, and DELETE newly reaches it, so a DELETE against another v2 catalog in a
session with these extensions loaded would hit a MatchError.

I also considered dropping the rowLineageAsDataCols workaround instead. That is not viable:
with it removed the lineage columns are no longer written, and TestCopyOnWriteWithLineage
fails with Structs do not match and ArrayIndexOutOfBoundsException rather than a plan
validation error. The workaround is still required.

Testing

No new test was needed. The existing testDelete reproduces this once validation is on, so
this enables spark.sql.planChangeValidation in ExtensionsTestBase. Spark gates per-rule
validation on the spark.testing system property, but the harness sets spark.testing
as a Spark conf, so validation was never actually running.

With validation enabled and without the fix, 30 tests fail across TestCopyOnWriteDelete
and TestCopyOnWriteWithLineage. With the fix the full spark-extensions suite passes:
2501 passed, 180 skipped, 0 failures (TestRewritePositionDeleteFilesProcedure hit an
unrelated flaky REST catalog 404 during initialization in one run and passes on its own).

Notes for reviewers

• The rule is not restricted to copy-on-write. RewriteUpdateTableForRowLineage and
  RewriteMergeIntoTableForRowLineage do not distinguish either, and I verified merge-on-read
  is unaffected (TestMergeOnReadWithLineage 40/40). Say the word if you would rather it were
  narrowed to the group-based path.
• Unlike the sibling rules, this one falls through to the unchanged plan instead of risking a
  MatchError, and guards on resolved the way Spark's own RewriteDeleteFromTable and
  RewriteMergeIntoTableForRowLineage do.
• Enabling plan validation for the whole extensions suite is deliberate, to stop this class of
  bug recurring, but it is the part of this change most likely to affect unrelated tests. Happy
  to scope it to the row lineage tests instead.

AI Disclosure

• Model: Claude Opus 5
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Investigate [Spark 3.5] V3 copy-on-write DELETE fails plan validation when spark.testing is enabled #18131, identify the root cause of the copy-on-write DELETE plan
  validation failure, and implement a fix with regression coverage. The agent reproduced the
  failure, traced it to the missing DELETE counterpart of the existing row lineage rules,
  empirically ruled out the alternative fix, and ran the full spark-extensions suite.