Releases: GoogleCloudPlatform/DataflowTemplates
Dataflow Templates 2025-10-07-00_RC00
Note: This release is being rolled out. It may not be in your region yet.
What's Changed
Improvements
- Fixed a performance regression in the Spanner change streams to BigQuery template that was introduced in the 2024-09-10-00_RC00.
- Only read schema when the col in mod is not in the stored schema. by @jiangzzhu in #2840
- Skip reading schema if the value capture mode is NEW_ROW_AND_OLD_VALUES. by @jiangzzhu in #2839
- feat: Add support for property graph on view by @jennyjiang98 in #2813
- Remove jinja variables option code by @derrickaw in #2826
New Contributors
- @jiangzzhu made their first contribution in #2840
Full Changelog: 2025-10-01-00_RC00...2025-10-07-00_RC00
Dataflow Templates 2025-10-01-00_RC00
What's Changed
Improvements
- Add Run, Staging and Validation stages for KafkaToBigquery Yaml template by @derrickaw in #2782
Bug fixes
- Fix vector index detection by @fbiville in #2151
- Change default CPU priority of bulk template to HIGH by @manitgupta in #2836
Package Upgrades
Known issues
Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00
version of the template due to a performance regression introduced in the 2024-09-10-00_RC00
release that might affect certain use-cases.
Full Changelog: 2025-09-23-00_RC00...2025-10-01-00_RC00
Dataflow Templates 2025-09-23-00_RC00
What's Changed
Improvements
- Introduce Schemaless graphs to import/export DDL by @smukil in #2749
- build(deps): bump org.awaitility:awaitility from 4.2.0 to 4.3.0 by @dependabot[bot] in #2794
- DataStreamToSpanner - Adding main table read at the beginning of main transaction by @darshan-sj in #2789
- CPU priority in reverse replication jobs by @bharadwaj-aditya in #2795
Known issues
Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00
version of the template due to a performance regression introduced in the 2024-09-10-00_RC00
release that might affect certain use-cases.
Full Changelog: 2025-09-16-00_RC00...2025-09-23-00_RC00
Dataflow Templates 2025-09-16-00_RC00
What's Changed
Improvements
- feat(datastream): add source type override for datastream cdc data by @liferoad in #2715
- Bump org.testcontainers:hivemq from 1.17.5 to 1.21.3 by @dependabot[bot] in #2729
- Fix CVE-2025-58056 by @stankiewicz in #2750
Package Upgrades
- chore(deps): update Python dependencies by @github-actions[bot] in #2778
Known issues
Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00
version of the template due to a performance regression introduced in the 2024-09-10-00_RC00
release that might affect certain use-cases.
Full Changelog: 2025-09-10-00_RC00...2025-09-16-00_RC00
Dataflow Templates 2025-09-10-00_RC00
What's Changed
Improvements
- Bump com.azure:azure-messaging-eventhubs from 5.15.0 to 5.21.0 by @dependabot[bot] in #2711
- Bump org.codehaus.mojo:exec-maven-plugin from 1.6.0 to 3.5.1 by @dependabot[bot] in #2710
- Datastream To SQL - Added Default Casing Parameter / Column Casing Functionality by @rahulkumar0630 in #2687
Bug fixes
- add float32 support to spanner-to-sourcedb by @VardhanThigle in #2722
- fix SourceDbToSpanner desc truncations by @derrickaw in #2734
- update docs and readme about template validation by @derrickaw in #2743
Known issues
Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00
version of the template due to a performance regression introduced in the 2024-09-10-00_RC00
release that might affect certain use-cases.
Full Changelog: 2025-09-02-00_RC00...2025-09-10-00_RC00
Dataflow Templates 2025-09-09-00_RC00
fix: update MongoDB container image tag in integration test setup (#2…
Dataflow Templates 2025-09-02-00_RC00
What's Changed
Improvements
- feat(import): add early index create threshold parameter by @liferoad in #2686
- [Yaml]: add dependency requirements etc by @derrickaw in #2694
- Bump org.twdata.maven:mojo-executor from 2.4.0 to 2.4.1 by @dependabot[bot] in #2700
- Bump org.codehaus.mojo:extra-enforcer-rules from 1.8.0 to 1.10.0 by @dependabot[bot] in #2699
- fix readme outputs for python and yaml modules by @derrickaw in #2704
Bug fixes
Full Changelog: 2025-08-26-00_RC00...2025-09-02-00_RC00
Dataflow Templates 2025-08-26-00_RC00
What's Changed
Improvements
- Bump com.google.apis:google-api-services-datastream from v1-rev20230902-2.0.0 to v1-rev20250722-2.0.0 by @dependabot[bot] in #2611
- build: update astra-io dependency to 4.18.1 by @liferoad in #2654
- build: add org.json dependency with secure version by @liferoad in #2661
- fix: enforce non-vulnerable version of okio to address by @liferoad in #2663
- fix: remove vendored Guava dependency to fix CVE-2023-2976 by @liferoad in #2669
- feat: enforce cypher 5 for generated queries by @nord-erik in #2620
Bug fixes
- Update FormatDatastreamRecordToJson.java by @dhercher in #2665
- Update kafka to bq README etc by @derrickaw in #2668
- revise metadata link and update docs by @derrickaw in #2662
New Contributors
- @nord-erik made their first contribution in #2620
Full Changelog: 2025-08-19-00_RC00...2025-08-26-00_RC00
Dataflow Templates 2025-08-19-00_RC00
What's Changed
Improvements
- Integration tests - Postgres to Postgres for Datastream to SQL Template by @rahulkumar0630 in #2617
- Upgrade Templates to Beam 2.67.0 by @Amar3tto in #2644
- Schema Map Support for Tables by @rahulkumar0630 in #2629
Full Changelog: 2025-08-12-00_RC00...2025-08-19-00_RC00
Dataflow Templates 2025-08-12-00_RC00
What's Changed
Improvements
- Replace session file usage with ISchemaMapper and add string overrides support by @Deep1998 in #2561
- Bump parquet-avro version to 1.15.2 by @jrmccluskey in #2606
Full Changelog: 2025-08-05-00_RC01...2025-08-12-00_RC00