[TEST ONLY] Trigger release integration validation#6993
Open
timothyw553 wants to merge 8 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Temporary PR to trigger release-validation integration tests requested for the UC / table-service validation work.
This PR is test-only and is not intended to merge into master. After the signal is collected, this PR should be closed.
What This Adds
.github/workflows/integration_test.yamlto run broader release-validation integration tests on PR CI.project/scripts/install-spark.shto download pinned Spark binary distributions with SHA512 verification.--spark-versionfiltering torun-integration-tests.pyso each CI job runs only the intended Spark runtime instead of all Spark variants in one runner.CI Coverage
The Integration Tests workflow now creates separate jobs/checks:
Scala + Python Integration / Spark 4.0.1_4.0Delta artifacts against Spark4.0.1Scala + Python Integration / Spark 4.1.0_4.1Delta artifacts against Spark4.1.0Scala + Python Integration / Spark 4.1.2_4.1Delta artifacts against Spark4.1.2Iceberg Integration / Spark 4.0.14.0.1and Iceberg1.10.1Each job sets
SPARK_VERSION/SPARK_MAJOR_MINORbefore UC setup, so the pinned UC build publishes the matching connector, for example:unitycatalog-spark_4.0_2.13for Spark4.0.1unitycatalog-spark_4.1_2.13for Spark4.1.0and4.1.24.1.2is a runtime patch-version check; it reuses the existing Spark4.1Delta artifact family.Related Local Validation
The separate UC table-service validation should be run locally with the focused UCDelta test pattern:
build/sbt 'sparkUnityCatalog/testOnly io.sparkuctest.UCDelta*'Do not run the broader
io.sparkuctest.*pattern for that validation, becauseUnityCatalogSupportTestresets the UC-related environment settings in-process.Current CI Run
https://github.com/delta-io/delta/actions/runs/27288371931