Releases · GoogleCloudDataproc/spark-bigquery-connector

Added new connector, spark-4.0-bigquery aimed to be used in Spark 4.0. Like Spark 4.0, this connector requires at
least Java 17 runtime. It is currently in preview mode.
PR #1367: Query Pushdown is no longer supported.
PR #1369: Catalog enhancements
PR #1374: Ensure TableId includes project ID if not explicitly set
PR #1376: materializationDataset is now optional to read from views or queries.
PR #1380: Fixed ImpersonatedCredentials serialization
PR #1381: Added the option to set custom credentials scopes
PR #1411: Added Support for SparkSession#executeCommand
BigQuery API has been upgraded to version 2.54.0
BigQuery Storage API has been upgraded to version 3.16.1
GAX has been upgraded to version 2.68.2
gRPC has been upgraded to version 1.74.0

@ddebowczyk92

@tom-s-powell

CVE-2025-24970, CVE-2025-25193: Upgrading netty to version 4.1.119.Final
PR #1284: Making BigQueryClientFactory Kryo serializable. Thanks @tom-s-powell !
PR #1345: materializationDataset is no longer needed to read from views or queries.

Issue #1254: fix getting partitioning fields for pseudo columns

Issue #1290: Stopped using metadata for optimized count path
Issue #1317: Improving OpenLineage 1.24.0+ compatibility
PR #1311: Improve read session expired error message
PR #1320: Set the temporaryGcsBucket to default to fs.gs.system.bucket if exists, negating the need to set it in Dataproc clusters.
BigQuery API has been upgraded to version 2.45.0
BigQuery Storage API has been upgraded to version 3.11.0
GAX has been upgraded to version 2.59.0
Netty has been upgraded to version 4.1.115.Final
Arrow has been upgraded to version 17.0.0
Avro has been upgraded to version 1.11.4

Releases: GoogleCloudDataproc/spark-bigquery-connector