Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: ensure snapshot files are stored with LF line endings by @lonless9 in #1123
• feat: remove session by @linhr in #1125
• fix: get optional Spark config by @linhr in #1126
• feat: more attributes and definitions for execution metrics by @linhr in #1119
• refactor: simplify actor actions by @linhr in #1128
• deps(cargo): bump the all group with 9 updates by @dependabot[bot] in #1129
• deps(npm): bump the all group with 10 updates by @dependabot[bot] in #1131
• deps(cargo): bump the breaking group with 3 updates by @dependabot[bot] in #1130
• feat: basic merge operation for Delta Lake by @lonless9 in #1093
• fix: remove SessionContext from codec by @linhr in #1134
• fix: ensure delta log json files include operation details and set up snapshot tests for Delta Lake by @lonless9 in #1133
• refactor: split driver logic by @linhr in #1135
• feat: set up metrics & enable checkpoint for Delta Lake by @lonless9 in #1136
• refactor: improve worker peer client management by @linhr in #1137
• feat: refactor functions datafusion 51 by @davidlghellin in #1105
• chore: various ops should not fail by @shehabgamin in #1140
• chore: update parquet config options by @shehabgamin in #1141
• fix: correct MergeIntoWriteNode Hash & avoid the side effect of explain statement by @lonless9 in #1139
• fix: ensure correct stats type for expression evaluator by @lonless9 in #1144
• feat: update proto and config for Spark 4.1 by @linhr in #1145
• fix: generate initial physical plan without physical optimizations to avoid misleading plan steps by @lonless9 in #1147
• fix: incompatible UDF serializer for Spark 4.1 by @linhr in #1148
• chore: prepare v0.4.5 by @linhr in #1143

Full Changelog: v0.4.4...v0.4.5

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• chore: revert the pin of hatch version and resolve ruff warnings by @lonless9 in #1090
• fix: assertion message by @YichiZhang0613 in #1098
• feat: make it possible to pass customised k8s labels to sail workers by @zemin-piao in #1103
• feat: support OpenTelemetry exporter and distributed tracing by @linhr in #1097
• deps(cargo): bump serde_with from 3.16.0 to 3.16.1 in the all group by @dependabot[bot] in #1101
• deps(npm): bump the all group with 4 updates by @dependabot[bot] in #1100
• deps(cargo): bump the breaking group with 2 updates by @dependabot[bot] in #1102
• refactor: invert dependency for data source registration by @lonless9 in #1099
• feat: new functions in datafusion 51 by @davidlghellin in #1106
• feat: query execution tracing by @linhr in #1104
• fix: handle Timestamp & TimestampNTZ Writes for Delta Lake by @lonless9 in #1095
• fix: resolve partition column mismatches and validate specs during writes by @lonless9 in #1108
• deps(cargo): bump the all group with 3 updates by @dependabot[bot] in #1112
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in #1113
• fix: refine shuffle consumption planning by @linhr in #1111
• feat: add PySpark function scanner & sail coverage by @keen85 in #1075
• refactor: use arrow take_record_batch by @gstvg in #1115
• feat: explain statement rework by @lonless9 in #1110
• feat: report execution metrics by @linhr in #1116
• fix: partition handling under column mapping for Delta Lake by @lonless9 in #1109
• fix: improve conflict checking and retries for concurrent Delta creation and blind appending by @lonless9 in #1117
• chore: address clippy 1.92.0 warnings by @lonless9 in #1118
• fix: ensure single partition input for GlobalLimitExec by @lonless9 in #1121
• chore: prepare v0.4.4 by @shehabgamin in #1120

New Contributors

• @YichiZhang0613 made their first contribution in #1098
• @keen85 made their first contribution in #1075
• @gstvg made their first contribution in #1115

Full Changelog: v0.4.3...v0.4.4

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• feat: add CI ibis test by @davidlghellin in #1033
• chore: apache license header by @shehabgamin in #1054
• fix: correct Apache License header by @shehabgamin in #1055
• fix: signum add Decimal128 by @davidlghellin in #1056
• chore: polish docs for clarity by @shehabgamin in #1058
• fix: max_by min_by by @davidlghellin in #1049
• docs: apollo.io integration by @linhr in #1064
• refactor: greatest and least by @davidlghellin in #1057
• chore: datafusion 51 by @lonless9 in #1031
• refactor: migrate to delta-kernel-rs foundation by @lonless9 in #1065
• chore: cargo-nextest by @lonless9 in #1070
• chore: skip flaky tests by @linhr in #1079
• deps(cargo): bump the all group with 9 updates by @dependabot[bot] in #1080
• deps(npm): bump the all group with 9 updates by @dependabot[bot] in #1083
• feat: add div to interval by @davidlghellin in #1077
• feat: allow EXPLAIN for SQL statements by @linhr in #1078
• feat: schema evolution for Iceberg by @lonless9 in #1048
• docs: minor updates by @linhr in #1085
• chore: update to delta-kernel-rs 0.18.1 arrow 57.1.0 by @lonless9 in #1084
• chore: pin hatch version to 1.15.1 in CI environment by @lonless9 in #1088
• refactor: better error handling for Delta Lake by @lonless9 in #1086
• chore: remove OS native SSL dependency by @linhr in #1087
• chore: prepare v0.4.3 by @shehabgamin in #1089

Full Changelog: v0.4.2...v0.4.3

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: ensure absolute paths for cross-engine Iceberg reads by @lonless9 in #1006
• refactor: consolidate redundant logic and improve reusability for Iceberg by @lonless9 in #1009
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in #1027
• deps(cargo): bump the all group with 5 updates by @dependabot[bot] in #1025
• fix: timestamp data type support for to_date function by @shehabgamin in #1024
• chore: update claude code by @shehabgamin in #1030
• feat: add try_avg Aggregate function by @davidlghellin in #1012
• chore: bump ibis test by @davidlghellin in #1029
• test: set up SQL BDD tests by @linhr in #1032
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in #1035
• chore: skip codecov report upload for dependabot by @linhr in #1038
• deps(cargo): bump the all group with 6 updates by @dependabot[bot] in #1036
• refactor: use datafusion 50 func by @davidlghellin in #909
• feat: time travel for Iceberg by @lonless9 in #1039
• feat: column mapping for Delta Lake by @lonless9 in #985
• feat: unity catalog by @shehabgamin in #1005
• feat: support try_ window functions by @linhr in #1040
• fix: remove the auto-initialize feature for PyO3 by @linhr in #1043
• chore: add workflow for ad-hoc multi-platform build by @linhr in #1044
• fix: multi-platform build inputs by @linhr in #1045
• fix: multi-platform build inputs by @linhr in #1046
• fix: multi-platform build inputs by @linhr in #1047
• fix: iceberg path handling on Windows platform by @lonless9 in #1042
• chore: prepare v0.4.2 by @linhr in #1041
• chore: prepare v0.4.2 by @linhr in #1051

Full Changelog: v0.4.1...v0.4.2

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• docs: fix typo in catalog guide by @linhr in #999
• feat: add try_sum Aggregate function by @davidlghellin in #960
• feat: partitioned write operation for Iceberg by @lonless9 in #1003
• chore: clippy 1.91 warnings by @lonless9 in #1004
• fix: only use rename table provider if necessary by @shehabgamin in #1008
• docs: partitioned Iceberg table guide by @linhr in #1016
• chore: prepare v0.4.1 by @linhr in #1022

Full Changelog: v0.4.0...v0.4.1

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• chore: update Rust toolchain installation commands by @linhr in #943
• deps(npm): bump @tailwindcss/vite from 4.1.13 to 4.1.14 by @dependabot[bot] in #953
• deps(npm): bump sql-formatter from 15.6.9 to 15.6.10 by @dependabot[bot] in #954
• deps(npm): bump @eslint/js from 9.36.0 to 9.37.0 by @dependabot[bot] in #955
• chore: update docs by @shehabgamin in #958
• fix: count_distinct(multiple, args) by @SparkApplicationMaster in #957
• deps(cargo): bump monostate from 1.0.0 to 1.0.1 by @dependabot[bot] in #952
• fix: fail fast for invalid gold test paths by @linhr in #942
• refactor: item taker by @linhr in #941
• feat: add --experimental_allow_proto3_optional when building proto by @zemin-piao in #959
• feat: syntax description for SQL AST by @linhr in #946
• refactor: share scan utilities for Delta Lake by @lonless9 in #921
• feat: get SQL text from AST by @linhr in #962
• chore: update dependencies by @linhr in #970
• fix: work around dependabot regression by @linhr in #971
• deps(cargo): bump the all group with 5 updates by @dependabot[bot] in #973
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in #972
• feat: iceberg rest catalog client generation by @shehabgamin in #961
• feat: basic read operation for Iceberg by @lonless9 in #944
• chore: execution configs by @shehabgamin in #975
• fix: correct subset logic in get_edges_contained_in_set by @davidlghellin in #969
• deps(npm): bump the all group with 7 updates by @dependabot[bot] in #978
• feat: iceberg arrow conversion by @shehabgamin in #987
• feat: iceberg rest catalog provider by @shehabgamin in #974
• Enrich HDFS doc with kerberos support info by @zemin-piao in #992
• deps(cargo): bump testcontainers from 0.25.0 to 0.25.1 in the all group by @dependabot[bot] in #990
• fix: iceberg catalog create_namespace by @shehabgamin in #993
• feat: basic write operation for Iceberg by @lonless9 in #976
• refactor: remove IcebergObjectStore by @lonless9 in #994
• chore: follow-up changes for Iceberg catalog by @linhr in #995
• fix: allow appending to non-existent Iceberg table by @lonless9 in #997
• docs: catalog guide and Iceberg guide by @linhr in #998
• chore: prepare v0.4.0 by @linhr in #996

New Contributors

• @zemin-piao made their first contribution in #959

Full Changelog: v0.3.7...v0.4.0

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• chore: update Cargo dependencies by @linhr in #937
• chore: update chumsky to 0.11 by @linhr in #938
• chore: update npm dependencies by @linhr in #939
• chore: deny macros used for development by @linhr in #940
• feat: df.unpivot method by @SparkApplicationMaster in #948
• chore: cargo deps by @shehabgamin in #949
• docs: add guide for SQL operators by @linhr in #947
• chore: prepare v0.3.7 by @shehabgamin in #950

Full Changelog: v0.3.6...v0.3.7

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• feat: add elt function by @davidlghellin in #820
• feat: inline and inline_outer funcs, fix: emty and all-null array datatype by @SparkApplicationMaster in #841
• feat: add try_parse_url function by @rafafrdz in #824
• feat: kustomize + support pod template for Sail workers by @shehabgamin in #833
• feat: stack function by @SparkApplicationMaster in #843
• fix: robots meta tag for docs by @shehabgamin in #845
• revert: robots meta tag for docs by @shehabgamin in #846
• fix: try_multiply function by @SparkApplicationMaster in #847
• deps(npm): bump the all group with 11 updates by @dependabot[bot] in #850
• deps(cargo): bump the all group with 6 updates by @dependabot[bot] in #851
• feat: add make_dt_interval function by @davidlghellin in #835
• chore: touch up docs by @shehabgamin in #852
• fix: session key without user ID by @linhr in #849
• refactor: handle side-effect of replaceWhere for Delta Lake by @lonless9 in #848
• feat: time travel for Delta Lake by @lonless9 in #854
• feat: binary file reader by @shehabgamin in #853
• chore: docs update by @linhr in #858
• feat: add version function by @rafafrdz in #855
• feat: months_between func, chore: refactor datetime funcs by @SparkApplicationMaster in #859
• feat: add user and session_user functions by @rafafrdz in #860
• chore: add basic function docs, missing functions, and aggregate functions to window list by @shehabgamin in #861
• fix: string escape by @linhr in #868
• chore: disable tracer by @shehabgamin in #869
• fix: map_from_arrays, from values column indices by @SparkApplicationMaster in #878
• fix: utc in timestamp literal by @shehabgamin in #872
• chore: use Claude Sonnet 4 for Claude Code by @linhr in #879
• docs: sql features, literals (update) by @everettroeth in #864
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in #882
• fix: exponential type parsing by @shehabgamin in #873
• chore: reorganize udf functions for consistency by @rafafrdz in #880
• feat: set up streaming events by @linhr in #832
• fix: fetch DuckDB arrow data in tests by @linhr in #884
• deps(cargo): bump the all group with 2 updates by @dependabot[bot] in #881
• feat: mcp batch create views by @shehabgamin in #875
• fix: avoid repartitioning child for PySpark UDTF by @linhr in #662
• feat: applyInArrow by @SparkApplicationMaster in #886
• refactor: readme touchup by @shehabgamin in #888
• docs: introduction (added word "multimodal") by @everettroeth in #889
• feat: cogrouped applyInArrow by @SparkApplicationMaster in #887
• fix: df.replace() method by @SparkApplicationMaster in #891
• fix: parsing real by @shehabgamin in #892
• fix: arrow-batched UDF with no args by @SparkApplicationMaster in #895
• docs: update SQL literal guide by @linhr in #896
• chore: add README images by @linhr in #897
• chore: optimize README by @linhr in #898
• deps(npm): bump eslint-plugin-vue from 10.4.0 to 10.5.0 by @dependabot[bot] in #903
• chore: upgrade Delta dependencies by @linhr in #899
• deps: upgrade eslint by @linhr in #906
• docs: sql features - data types added by @everettroeth in #893
• chore: datafusion 50 by @shehabgamin in #799
• refactor: move functions to a separate crate by @linhr in #908
• refactor: remove hard PyO3 dependency for error cause by @linhr in #910
• refactor: move plan extensions by @linhr in #912
• fix: approx_count_distinct return type by @SparkApplicationMaster in #914
• fix: sorting by aggregate expressions by @linhr in #915
• feat: delete operation for Delta Lake by @lonless9 in #856
• chore: set up config for join reorder optimizer by @shehabgamin in #810
• fix: handle side-effect of delete operation for Delta Lake by @lonless9 in #916
• docs: update data type guides by @linhr in #918
• fix: map_from_entries wrong result on null input by @SparkApplicationMaster in #920
• fix: explicitly find and pass llvm-profdata path to grcov by @lonless9 in #923
• docs: identifiers in SQL by @linhr in #922
• fix: map_from_entries error on lists with null entries by @SparkApplicationMaster in #926
• feat: implement join reorder physical optimizer with the DPHyp algorithm by @lonless9 in #917
• docs: improve intro page order by @linhr in #924
• feat: file metadata cache for Parquet by @linhr in #928
• chore: prepare v0.3.6 by @linhr in #929

New Contributors

• @everettroeth made their first contribution in #864

Full Changelog: v0.3.5...v0.3.6

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: changelog by @linhr in #821
• fix: changelog by @linhr in #823
• fix: empty map type in from values, explode map with null values by @SparkApplicationMaster in #829
• feat: set up sail-physical-optimizer by @lonless9 in #831
• fix: writing Delta table with partitioned inputs by @linhr in #837
• chore: prepare v0.3.5 by @linhr in #838

Full Changelog: v0.3.4...v0.3.5

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• feat: add try_mod function Spark 4.0.0 by @davidlghellin in #727
• feat: add make_interval function by @davidlghellin in #682
• fix: adjust default session timeout and update docs by @linhr in #761
• fix: reading and writing compressed files by @shehabgamin in #760
• fix: nth_value failure on ignoreNulls=true by @SparkApplicationMaster in #767
• fix: withColumnsRenamed result if chained renames in one input by @SparkApplicationMaster in #764
• fix: window functions inconsistent uint64 return type by @SparkApplicationMaster in #765
• fix: median return type to double by @SparkApplicationMaster in #769
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in #771
• deps(cargo): bump the all group with 4 updates by @dependabot[bot] in #770
• refactor: split plan resolver by @linhr in #772
• feat: map_entries func, map keys deduplication logic by @SparkApplicationMaster in #774
• chore: code coverage by @lonless9 in #758
• refactor: split expression resolver by @linhr in #775
• fix: predicate negation for SQL by @linhr in #776
• feat: map_from_entries, unify all ways to create a map in MapFunction by @SparkApplicationMaster in #777
• chore: add url codecov in README by @davidlghellin in #778
• feat: split function by @rafafrdz in #722
• feat: text file format by @shehabgamin in #737
• feat: map_concat func, args check for map(k, v) func by @SparkApplicationMaster in #779
• refactor: split data source options resolver by @linhr in #781
• chore: codecov config by @lonless9 in #780
• feat: support DeltaPlan for Delta Lake by @lonless9 in #768
• chore: update data format docs by @shehabgamin in #783
• fix: element_at and try_element_at inconsistent error treatment, chore: simplify, unify by @SparkApplicationMaster in #785
• chore: refactor array and collection modules by @SparkApplicationMaster in #786
• docs: update unsupported SQL features by @linhr in #788
• fix: upload codecov report after tests finished by @lonless9 in #782
• feat: str_to_map function by @SparkApplicationMaster in #787
• feat: add width_bucket function by @davidlghellin in #762
• fix: position and locate funcs with startpos arg by @SparkApplicationMaster in #789
• fix: projection optimization for extension logical nodes by @linhr in #790
• deps(npm): bump the all group with 4 updates by @dependabot[bot] in #791
• deps(cargo): bump the all group with 7 updates by @dependabot[bot] in #792
• fix: split func type coersion, perf: split, str_to_map with scalar args by @SparkApplicationMaster in #795
• chore: update cargo deps by @shehabgamin in #796
• feat: regex_instr, fix: string funcs type casting, chore: string funcs refactoring by @SparkApplicationMaster in #798
• feat: support Spark Connect streaming API by @linhr in #751
• feat: json_object_keys, fix: get_json_object with scalar by @SparkApplicationMaster in #801
• refactor: kernel log replay for Delta Lake by @lonless9 in #794
• feat: collect_list(cols) over (window_frame) by @SparkApplicationMaster in #806
• refactor: use topandas in derived tpch example by @shehabgamin in #807
• fix: data sink exec by @shehabgamin in #813
• fix: deterministic schema for withColumns by @linhr in #814
• fix: handle harmless OSError in socket test by @lonless9 in #815
• deps(npm): bump @ianvs/prettier-plugin-sort-imports from 4.6.2 to 4.7.0 in the all group by @dependabot[bot] in #817
• feat: heartbeat for long-running execution by @linhr in #818
• chore: prepare v0.3.4 by @linhr in #819

Full Changelog: v0.3.3...v0.3.4