Modernize/arrow go upgrade - #2
Merged
Merged
Conversation
Modernize the ~6-year-stale build: bump go.mod from go 1.14 to go 1.25 and
migrate the Apache Arrow dependency from the EOL module
github.com/apache/arrow/go/arrow v0.0.0-20200711... to the current
github.com/apache/arrow-go/v18 v18.7.0 (the project moved to apache/arrow-go).
Import path migration (all .go and .tmpl):
github.com/apache/arrow/go/arrow -> github.com/apache/arrow-go/v18/arrow
API reconciliation for Arrow v18 (old -> new):
- array.Interface -> arrow.Array
- array.Column -> arrow.Column
- array.Table -> arrow.Table
- array.Record -> arrow.RecordBatch (Record is a deprecated alias)
- array.NewChunked/NewColumn -> arrow.NewChunked/arrow.NewColumn
- array.NewTableReader -> unchanged (still in the array package)
Behavior-preserving reimplementations for APIs removed in Arrow v18:
- Column.NewSlice was removed. Added sliceColumn() (pkg/dataframe/mutations.go)
which slices each chunk of a Column's Chunked data via array.NewSlice and
rebuilds the Column, with correct ref-counting (NewChunked/NewColumn retain).
- arrow.Table now requires AddColumn(pos, f, c) (Table, error) and embeds
fmt.Stringer. Added tableReaderFacade.AddColumn (builds a new *array.Table
from the facade's columns + new column at pos) and String() to satisfy the
interface. array.NewTableReader is unchanged so the existing read path
(Display) is unaffected.
Documented Arrow-v18 rendering changes reflected in updated expectations:
- MonthInterval now JSON-marshals as {"months": N} (was a bare int).
- Decimal128 array display now respects column precision (e.g. precision 1
renders 1.84e9 as "2e+09"); previously it printed the raw {lo hi} struct.
- Nested type String() now appends ", nullable" for nullable fields
(Arrow Go v18.6.0 "add nullability to struct type String()").
Verification:
- go build ./... : clean
- go vet ./... : clean
- make build : regenerates all .gen.go from templates (goimports)
- make test (-tags=assert) : all packages pass
- make ci (-tags='debug assert') : all packages pass
- go test -race -count=1 ./... : all packages pass
Codegen remains reproducible: regenerated .gen.go match the .tmpl sources.
…decimal test Addresses review findings on the v18 upgrade commit (bc13096). tableReaderFacade.AddColumn now mirrors Apache Arrow for Go's own simpleTable.AddColumn contract instead of a naive rebuild: - Validate col.Len() == NumRows() and return arrow.ErrInvalid on mismatch (previously a wrong-length column could panic inside array.NewTable). - Validate arrow.TypeEqual(field.Type, col.DataType()) and return an error on mismatch (previously a type mismatch could panic). - Derive the new schema via Schema.AddField(pos, field) so the original schema metadata and endianness are preserved (previously NewSchema(fields, nil) silently dropped metadata and reset endianness). - Position bounds still checked and reported as an error. Add pkg/dataframe/tablefacade_test.go covering: insert at middle, schema metadata preservation, length-mismatch error, type-mismatch error, and position-out-of-bounds error. All run under memory.CheckedAllocator to catch ref-count leaks. Add TestSmartBuilderDecimal128StoredValues to assert the exact stored decimal128.Num values via *array.Decimal128, independent of Arrow's precision-dependent display formatting. The generated string assertion collapses distinct values at low precision (i=6,7,8 all render "1e+10"), so it alone could not prove SmartBuilder stored the correct values. Verification: - go vet ./... : clean - make test (-tags=assert) : all packages pass - make ci (-tags='debug assert') : all packages pass - go test -race -count=1 ./... : all packages pass
Close the first Arrow data-type gap. Decimal256 now flows through the same
Object/collection/iterator/dataframe/smartbuilder codegen path as Decimal128.
tmpldata (pkg/object/objects.tmpldata):
- New Decimal256 entry mirroring Decimal128's structure. Comparison uses
decimal256.Num's native Less/Greater/Cmp/LessEqual/GreaterEqual methods.
- name="decimal256" to match arrow.Decimal256Type.Name() (used for the
smartbuilder test column name).
- MaxValue = 2^255 - 1 (the true max signed 256-bit value; arrow has no
MaxDecimal256 constant).
- Full CastTo wiring: Decimal256 converts From {Boolean, Int64, Uint64,
Decimal128 (via FromDecimal128), decimal256.Num, Decimal256} with real
conversions, and From other types as NotImplemented. Cross-type From:Decimal256
entries added to all other 21 types, mirroring Decimal128 exactly including
the Overflow flags on Int64/Uint64 so Checked conversions detect overflow.
- TestConstructor/TestTypes use decimal256.FromI64 with Precision:38 Scale:0
so values render as clean integers in the smartbuilder test.
Hand-written code (pkg/object/decimal.go):
- NewDecimal256FromInt64 / NewDecimal256FromU64 (used by codegen casts).
- toI64/toU64/LowBits/Sign accessors (used by Decimal256 -> primitive casts).
- MarshalJSON/UnmarshalJSON encoding the four little-endian limbs (l0..l3),
mirroring the lo/hi encoding used for Decimal128 and enabling round-trips.
iterator (pkg/iterator/asjson.go):
- decimal256AsJSON mirroring decimal128AsJSON.
Templates: added the decimal256 import to the six templates that hardcode the
decimal128 import (object.gen.go.tmpl, object.gen._test.go.tmpl,
valueiterator.gen.go.tmpl, chunkiterator.gen.go.tmpl, element_numeric.gen.go.tmpl,
smartbuilder_test_data.gen.go.tmpl) so generated code resolves the package.
Tests (pkg/object/decimal_test.go):
- Construction, toI64/toU64 helpers, signed comparison (Eq/Less/Greater/etc),
JSON round-trip (including MaxDecimal256), and CastToDecimal256 from
Int64/Uint64/Boolean. Plus the codegen-generated Decimal256 comparison and
Checked-conversion tests.
Verification:
- go vet ./... : clean
- make build : regenerates cleanly, no drift
- make test (-tags=assert) : all packages pass
- make ci (-tags='debug assert') : all packages pass
- go test -race -count=1 ./... : all packages pass
Close the byte/string data-type gaps. The library now supports 26 Arrow data
types (was 22): added Binary, FixedSizeBinary ([]byte), and LargeString (string).
tmpldata (pkg/object/objects.tmpldata):
- Binary / FixedSizeBinary: Type []byte, lexicographic comparison via
bytes.Compare/bytes.Equal, casts From {[]byte, Binary/String}, cross-type
From entries (String + Boolean implemented, rest NotImplemented).
- LargeString: Type string, mirrors String (casts From {string, String}).
- All three ExcludeGenerate valueiterator/chunkiterator (array.Binary etc.
expose Value(i) but no bulk Values(), so they need hand-written iterators).
- LargeBinary was intentionally NOT added: arrow-go provides no
LargeBinaryBuilder (LargeBinary arrays are built via the generic
BinaryBuilder parameterized by dtype), so it does not fit the {Name}Builder
codegen pattern.
Hand-written value iterators (pkg/iterator/):
- binaryvalueiterator.go, fixedsizebinaryvalueiterator.go,
largestringvalueiterator.go — mirror stringvalueiterator.go, using the
generic ChunkIterator and per-index Value(i).
- AsJSON helpers (binaryAsJSON, fixedSizeBinaryAsJSON, largeStringAsJSON) in
asjson.go.
Template fixes (pkg/object/object.gen._test.go.tmpl):
- CastableTo/MethodTo tests now use reflect.DeepEqual instead of `!=` so
slice-based Object types (Binary/FixedSizeBinary) can be compared.
- The String-target special case now yields "false" only for Boolean sources
and "" for all others (previously hardcoded "false" for every non-String
source, which only happened to be correct for Boolean).
Tests (pkg/object/binary_test.go):
- Construction, lexicographic comparison, casts from primitives, and
String<->Binary round-trip for all three new types.
Verification:
- go vet ./... clean; go build ./... clean
- make build regenerates cleanly (no drift)
- make test (-tags=assert), make ci (-tags='debug assert'), go test -race all pass
Close another scalar data-type gap. The library now supports 27 Arrow data
types: added MonthDayNanoInterval (arrow.MonthDayNanoInterval{Months, Days,
Nanoseconds}), the third interval type alongside DayTimeInterval and
MonthInterval.
tmpldata (pkg/object/objects.tmpldata):
- MonthDayNanoInterval entry mirroring DayTimeInterval's structure. Comparison
uses arrow.MonthDayNanoInterval.Cmp() for correct lexicographic ordering of
(Months, Days, Nanoseconds) rather than error-prone manual field logic.
- CastTo wires From {arrow.MonthDayNanoInterval, MonthDayNanoInterval
(identity), Boolean}; all other conversions NotImplemented.
- Cross-type From:MonthDayNanoInterval entries added to all types: Boolean
references the correct fields (Months/Days/Nanoseconds), String uses %#v,
the rest NotImplemented. The DayTimeInterval <-> MonthDayNanoInterval
cross-conversion is NotImplemented because the structs are not directly
convertible.
iterator (pkg/iterator/asjson.go):
- monthDayNanoIntervalAsJSON mirroring dayTimeIntervalAsJSON.
Tests (pkg/object/monthdaynanointerval_test.go):
- Construction, lexicographic comparison (including day-field tiebreak), and
casts from the native type and Boolean.
Verification:
- go vet ./... clean; go build ./... clean
- make build regenerates cleanly (no drift)
- make test, make ci, go test -race all pass
…mal256 cast/JSON, binary base64) Addresses findings from the data-type review. BLOCKER — CastElement (pkg/dataframe/element.go): The hand-written CastElement dispatch only covered the original numeric/date types, so DataFrame equality/comparison operations panicked for every other type — including all 5 newly-added types (Decimal256, Binary, FixedSizeBinary, LargeString, MonthDayNanoInterval) plus pre-existing gaps (Boolean, Decimal128, String, Timestamp, Duration, Float16, Time32/64, MonthInterval, DayTimeInterval). Added dispatch cases for every type that has a generated Element implementation. Added TestCastElementCoversAllTypes as a regression test covering all 16 reachable types. MAJOR — CastToDecimal256 native type (pkg/object/objects.tmpldata): Decimal256's CastTo was missing a From:decimal256.Num entry, so CastToDecimal256(decimal256.Num) returned false and SmartBuilder rejected native Arrow decimal256 values. Added the native identity cast. Added TestCastToDecimal256Native. MAJOR — Decimal256 UnmarshalJSON null panic (pkg/object/decimal.go): Unmarshaling JSON null into a Decimal256 nil-dereferenced because aux was a pointer. Now unmarshals into a struct value so null decodes to the zero Decimal256 without panicking. Added TestDecimal256JSONNull. MAJOR — Binary/FixedSizeBinary JSON serialization (pkg/iterator/asjson.go): binaryAsJSON/fixedSizeBinaryAsJSON returned string(v.([]byte)), which silently corrupts non-UTF-8 bytes during JSON encoding. Now returns the raw []byte so encoding/json base64-encodes it, matching Apache Arrow's binary JSON serialization (lossless round-trip). Verification: - go vet ./... clean; go build ./... clean - make test, make ci, go test -race all pass
Add the Arrow 14+ view types, bringing the total to 29 supported data types.
tmpldata (pkg/object/objects.tmpldata):
- StringView: Type string, mirrors String/LargeString. name="string_view"
to match arrow.StringViewType.Name().
- BinaryView: Type []byte, mirrors Binary. name="binary_view".
Lexicographic comparison via bytes.Compare/bytes.Equal.
- Both ExcludeGenerate valueiterator/chunkiterator (array.StringView/
BinaryView expose Value(i) but no bulk Values() accessor).
Hand-written value iterators (pkg/iterator/):
- stringviewvalueiterator.go, binaryviewvalueiterator.go — mirror the
String/Binary iterators against the generic ChunkIterator.
- stringViewAsJSON / binaryViewAsJSON in asjson.go (binary returns raw
[]byte for base64 JSON encoding).
CastElement (pkg/dataframe/element.go):
- Added *arrow.StringViewType and *arrow.BinaryViewType dispatch cases so
DataFrame equality/comparison operations don't panic for view columns.
- Extended TestCastElementCoversAllTypes to cover both view types.
Verification:
- go vet ./... clean; go build ./... clean
- make build regenerates cleanly (no drift)
- make test, make ci, go test -race all pass
…rcleCI The existing GitHub Actions workflow pinned go-version: '1.20', which cannot build the module after the upgrade (go.mod now requires go 1.25). The check run on PR #2 failed in ~16s for exactly this reason. CircleCI (.circleci/config.yml) is long-obsolete (golang:1.14.4) and superseded by GitHub Actions. Changes: - .github/workflows/go.yml: use actions/setup-go@v5 with go-version-file: go.mod so CI always tracks the module's declared Go version (currently 1.25.0); install goimports (required by make build's codegen to format generated .gen.go files); run make build, make test, and make ci in distinct steps mirroring the local developer flow. Enable module cache. - Remove .circleci/ entirely (obsolete, references Go 1.14.4). Verified locally: `make clean && make build` regenerates all codegen from scratch with no drift, and `make test` / `make ci` both pass.
The previous commit removed the obsolete CircleCI config but the workflow modernization didn't get staged with it. This completes the CI update: actions/setup-go@v5 with go-version-file: go.mod (so CI always tracks the module's Go version, currently 1.25.0) plus an explicit goimports install (required by make build's codegen) and make build / make test / make ci steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modernizes this ~6-year-stale library to current Go and Apache Arrow Go, and closes the Arrow data-type coverage gaps for the scalar types that flow through the codegen pipeline. Brings supported types from 22 → 29.
Every direct code change is covered by tests; two independent reviewer passes were run.
Go & Arrow upgrade
go.mod:go 1.14→go 1.25.0github.com/apache/arrow/go/arrow v0.0.0-20200711...→ currentgithub.com/apache/arrow-go/v18 v18.7.0(the project moved to theapache/arrow-gorepo; module path is now.../v18/arrow)..goand.tmplsources.array.Interface→arrow.Array,array.Column→arrow.Column,array.Table→arrow.Table,array.Record→arrow.RecordBatcharray.NewChunked/NewColumn→arrow.*Column.NewSliceassliceColumn()(per-chunkarray.NewSlicewith balanced ref-counting)arrow.Tableinterface requirements (AddColumn+fmt.Stringer) ontableReaderFacadeNew data types (22 → 29)
Added via the existing codegen pipeline (
objects.tmpldata→.tmpl→.gen.go), so each new type gets full Object/conversion/iterator/builder/dataframe-element coverage:Decimal256decimal256.NumLess/Greater/Cmp; 4-limb JSON round-tripBinary,FixedSizeBinary[]byte, lexicographic viabytes.CompareLargeStringStringMonthDayNanoIntervalCmp(Months, Days, Nanoseconds)StringView,BinaryViewComposite types (
List/Struct/FixedSizeList) were already handled by hand-written iterators and are unaffected.LargeBinaryis intentionally not added — arrow-go has no dedicatedLargeBinaryBuilder(built via the genericBinaryBuilder), so it doesn't fit the{Name}Buildercodegen pattern.Review-driven fixes
Two reviewer passes caught real issues, all fixed with regression tests:
CastElementwas missing ~16 types (all new + pre-existing Boolean/Decimal128/String/etc.) → DataFrame ops would panic. Added dispatch for all types +TestCastElementCoversAllTypes.CastToDecimal256missing the nativedecimal256.Numentry.Decimal256.UnmarshalJSONnil-pointer panic on JSONnull.Binary/FixedSizeBinaryJSON serialization now lossless (base64, matching Arrow).tableReaderFacade.AddColumnmade faithful to Arrow's contract (length/type validation, schema-metadata preservation).Behavior changes (documented)
Three test expectations changed to reflect upstream Arrow v18 behavior (verified against arrow source; stored values unchanged — representational only):
MonthIntervalJSON now marshals as{"months": N}(was a bare int)Decimal128array display now respects column precision (e.g. precision 1 renders2e+09for1.84e9)String()now appends, nullablefor nullable list elementsCI
.circleci/(Go 1.14.4).github/workflows/go.yml:actions/setup-go@v5withgo-version-file: go.mod(auto-tracks the module's Go version), explicitgoimportsinstall for codegen, andmake build/make test/make cistepsVerification
go vet ./...cleango build ./...cleanmake buildregenerates codegen from scratch with no driftmake test(-tags=assert) — all packages passmake ci(-tags='debug assert') — all packages passgo test -race -count=1 ./...— cleanbuild-and-testcheck on this branch: ✅ passing