Skip to content

Modernize/arrow go upgrade - #2

Merged
nickpoorman merged 10 commits into
masterfrom
modernize/arrow-go-upgrade
Jul 28, 2026
Merged

Modernize/arrow go upgrade#2
nickpoorman merged 10 commits into
masterfrom
modernize/arrow-go-upgrade

Conversation

@nickpoorman

@nickpoorman nickpoorman commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

Modernizes this ~6-year-stale library to current Go and Apache Arrow Go, and closes the Arrow data-type coverage gaps for the scalar types that flow through the codegen pipeline. Brings supported types from 22 → 29.

Every direct code change is covered by tests; two independent reviewer passes were run.

Go & Arrow upgrade

  • go.mod: go 1.14go 1.25.0
  • Arrow: EOL github.com/apache/arrow/go/arrow v0.0.0-20200711... → current github.com/apache/arrow-go/v18 v18.7.0 (the project moved to the apache/arrow-go repo; module path is now .../v18/arrow).
  • Migrated all 76 import references across .go and .tmpl sources.
  • Reconciled every Arrow v18 API breakage:
    • array.Interfacearrow.Array, array.Columnarrow.Column, array.Tablearrow.Table, array.Recordarrow.RecordBatch
    • array.NewChunked/NewColumnarrow.*
    • Reimplemented the removed Column.NewSlice as sliceColumn() (per-chunk array.NewSlice with balanced ref-counting)
    • Implemented the new arrow.Table interface requirements (AddColumn + fmt.Stringer) on tableReaderFacade

New data types (22 → 29)

Added via the existing codegen pipeline (objects.tmpldata.tmpl.gen.go), so each new type gets full Object/conversion/iterator/builder/dataframe-element coverage:

Type Notes
Decimal256 native decimal256.Num Less/Greater/Cmp; 4-limb JSON round-trip
Binary, FixedSizeBinary []byte, lexicographic via bytes.Compare
LargeString mirrors String
MonthDayNanoInterval native Cmp (Months, Days, Nanoseconds)
StringView, BinaryView Arrow 14+ view types

Composite types (List/Struct/FixedSizeList) were already handled by hand-written iterators and are unaffected. LargeBinary is intentionally not added — arrow-go has no dedicated LargeBinaryBuilder (built via the generic BinaryBuilder), so it doesn't fit the {Name}Builder codegen pattern.

Review-driven fixes

Two reviewer passes caught real issues, all fixed with regression tests:

  • BLOCKER: CastElement was missing ~16 types (all new + pre-existing Boolean/Decimal128/String/etc.) → DataFrame ops would panic. Added dispatch for all types + TestCastElementCoversAllTypes.
  • CastToDecimal256 missing the native decimal256.Num entry.
  • Decimal256.UnmarshalJSON nil-pointer panic on JSON null.
  • Binary/FixedSizeBinary JSON serialization now lossless (base64, matching Arrow).
  • tableReaderFacade.AddColumn made faithful to Arrow's contract (length/type validation, schema-metadata preservation).

Behavior changes (documented)

Three test expectations changed to reflect upstream Arrow v18 behavior (verified against arrow source; stored values unchanged — representational only):

  • MonthInterval JSON now marshals as {"months": N} (was a bare int)
  • Decimal128 array display now respects column precision (e.g. precision 1 renders 2e+09 for 1.84e9)
  • Nested type String() now appends , nullable for nullable list elements

CI

  • Removed obsolete .circleci/ (Go 1.14.4)
  • Modernized .github/workflows/go.yml: actions/setup-go@v5 with go-version-file: go.mod (auto-tracks the module's Go version), explicit goimports install for codegen, and make build / make test / make ci steps

Verification

  • go vet ./... clean
  • go build ./... clean
  • make build regenerates codegen from scratch with no drift
  • make test (-tags=assert) — all packages pass
  • make ci (-tags='debug assert') — all packages pass
  • go test -race -count=1 ./... — clean
  • GitHub Actions build-and-test check on this branch: ✅ passing

Modernize the ~6-year-stale build: bump go.mod from go 1.14 to go 1.25 and
migrate the Apache Arrow dependency from the EOL module
github.com/apache/arrow/go/arrow v0.0.0-20200711... to the current
github.com/apache/arrow-go/v18 v18.7.0 (the project moved to apache/arrow-go).

Import path migration (all .go and .tmpl):
  github.com/apache/arrow/go/arrow -> github.com/apache/arrow-go/v18/arrow

API reconciliation for Arrow v18 (old -> new):
  - array.Interface            -> arrow.Array
  - array.Column               -> arrow.Column
  - array.Table                -> arrow.Table
  - array.Record               -> arrow.RecordBatch  (Record is a deprecated alias)
  - array.NewChunked/NewColumn -> arrow.NewChunked/arrow.NewColumn
  - array.NewTableReader       -> unchanged (still in the array package)

Behavior-preserving reimplementations for APIs removed in Arrow v18:
  - Column.NewSlice was removed. Added sliceColumn() (pkg/dataframe/mutations.go)
    which slices each chunk of a Column's Chunked data via array.NewSlice and
    rebuilds the Column, with correct ref-counting (NewChunked/NewColumn retain).
  - arrow.Table now requires AddColumn(pos, f, c) (Table, error) and embeds
    fmt.Stringer. Added tableReaderFacade.AddColumn (builds a new *array.Table
    from the facade's columns + new column at pos) and String() to satisfy the
    interface. array.NewTableReader is unchanged so the existing read path
    (Display) is unaffected.

Documented Arrow-v18 rendering changes reflected in updated expectations:
  - MonthInterval now JSON-marshals as {"months": N} (was a bare int).
  - Decimal128 array display now respects column precision (e.g. precision 1
    renders 1.84e9 as "2e+09"); previously it printed the raw {lo hi} struct.
  - Nested type String() now appends ", nullable" for nullable fields
    (Arrow Go v18.6.0 "add nullability to struct type String()").

Verification:
  - go build ./...        : clean
  - go vet ./...          : clean
  - make build            : regenerates all .gen.go from templates (goimports)
  - make test  (-tags=assert)            : all packages pass
  - make ci    (-tags='debug assert')    : all packages pass
  - go test -race -count=1 ./...         : all packages pass

Codegen remains reproducible: regenerated .gen.go match the .tmpl sources.
…decimal test

Addresses review findings on the v18 upgrade commit (bc13096).

tableReaderFacade.AddColumn now mirrors Apache Arrow for Go's own
simpleTable.AddColumn contract instead of a naive rebuild:
  - Validate col.Len() == NumRows() and return arrow.ErrInvalid on mismatch
    (previously a wrong-length column could panic inside array.NewTable).
  - Validate arrow.TypeEqual(field.Type, col.DataType()) and return an error
    on mismatch (previously a type mismatch could panic).
  - Derive the new schema via Schema.AddField(pos, field) so the original
    schema metadata and endianness are preserved (previously NewSchema(fields, nil)
    silently dropped metadata and reset endianness).
  - Position bounds still checked and reported as an error.

Add pkg/dataframe/tablefacade_test.go covering: insert at middle, schema
metadata preservation, length-mismatch error, type-mismatch error, and
position-out-of-bounds error. All run under memory.CheckedAllocator to catch
ref-count leaks.

Add TestSmartBuilderDecimal128StoredValues to assert the exact stored
decimal128.Num values via *array.Decimal128, independent of Arrow's
precision-dependent display formatting. The generated string assertion
collapses distinct values at low precision (i=6,7,8 all render "1e+10"),
so it alone could not prove SmartBuilder stored the correct values.

Verification:
  - go vet ./...                          : clean
  - make test  (-tags=assert)             : all packages pass
  - make ci    (-tags='debug assert')     : all packages pass
  - go test -race -count=1 ./...          : all packages pass
Close the first Arrow data-type gap. Decimal256 now flows through the same
Object/collection/iterator/dataframe/smartbuilder codegen path as Decimal128.

tmpldata (pkg/object/objects.tmpldata):
  - New Decimal256 entry mirroring Decimal128's structure. Comparison uses
    decimal256.Num's native Less/Greater/Cmp/LessEqual/GreaterEqual methods.
  - name="decimal256" to match arrow.Decimal256Type.Name() (used for the
    smartbuilder test column name).
  - MaxValue = 2^255 - 1 (the true max signed 256-bit value; arrow has no
    MaxDecimal256 constant).
  - Full CastTo wiring: Decimal256 converts From {Boolean, Int64, Uint64,
    Decimal128 (via FromDecimal128), decimal256.Num, Decimal256} with real
    conversions, and From other types as NotImplemented. Cross-type From:Decimal256
    entries added to all other 21 types, mirroring Decimal128 exactly including
    the Overflow flags on Int64/Uint64 so Checked conversions detect overflow.
  - TestConstructor/TestTypes use decimal256.FromI64 with Precision:38 Scale:0
    so values render as clean integers in the smartbuilder test.

Hand-written code (pkg/object/decimal.go):
  - NewDecimal256FromInt64 / NewDecimal256FromU64 (used by codegen casts).
  - toI64/toU64/LowBits/Sign accessors (used by Decimal256 -> primitive casts).
  - MarshalJSON/UnmarshalJSON encoding the four little-endian limbs (l0..l3),
    mirroring the lo/hi encoding used for Decimal128 and enabling round-trips.

iterator (pkg/iterator/asjson.go):
  - decimal256AsJSON mirroring decimal128AsJSON.

Templates: added the decimal256 import to the six templates that hardcode the
decimal128 import (object.gen.go.tmpl, object.gen._test.go.tmpl,
valueiterator.gen.go.tmpl, chunkiterator.gen.go.tmpl, element_numeric.gen.go.tmpl,
smartbuilder_test_data.gen.go.tmpl) so generated code resolves the package.

Tests (pkg/object/decimal_test.go):
  - Construction, toI64/toU64 helpers, signed comparison (Eq/Less/Greater/etc),
    JSON round-trip (including MaxDecimal256), and CastToDecimal256 from
    Int64/Uint64/Boolean. Plus the codegen-generated Decimal256 comparison and
    Checked-conversion tests.

Verification:
  - go vet ./...                          : clean
  - make build                            : regenerates cleanly, no drift
  - make test  (-tags=assert)             : all packages pass
  - make ci    (-tags='debug assert')     : all packages pass
  - go test -race -count=1 ./...          : all packages pass
Close the byte/string data-type gaps. The library now supports 26 Arrow data
types (was 22): added Binary, FixedSizeBinary ([]byte), and LargeString (string).

tmpldata (pkg/object/objects.tmpldata):
  - Binary / FixedSizeBinary: Type []byte, lexicographic comparison via
    bytes.Compare/bytes.Equal, casts From {[]byte, Binary/String}, cross-type
    From entries (String + Boolean implemented, rest NotImplemented).
  - LargeString: Type string, mirrors String (casts From {string, String}).
  - All three ExcludeGenerate valueiterator/chunkiterator (array.Binary etc.
    expose Value(i) but no bulk Values(), so they need hand-written iterators).
  - LargeBinary was intentionally NOT added: arrow-go provides no
    LargeBinaryBuilder (LargeBinary arrays are built via the generic
    BinaryBuilder parameterized by dtype), so it does not fit the {Name}Builder
    codegen pattern.

Hand-written value iterators (pkg/iterator/):
  - binaryvalueiterator.go, fixedsizebinaryvalueiterator.go,
    largestringvalueiterator.go — mirror stringvalueiterator.go, using the
    generic ChunkIterator and per-index Value(i).
  - AsJSON helpers (binaryAsJSON, fixedSizeBinaryAsJSON, largeStringAsJSON) in
    asjson.go.

Template fixes (pkg/object/object.gen._test.go.tmpl):
  - CastableTo/MethodTo tests now use reflect.DeepEqual instead of `!=` so
    slice-based Object types (Binary/FixedSizeBinary) can be compared.
  - The String-target special case now yields "false" only for Boolean sources
    and "" for all others (previously hardcoded "false" for every non-String
    source, which only happened to be correct for Boolean).

Tests (pkg/object/binary_test.go):
  - Construction, lexicographic comparison, casts from primitives, and
    String<->Binary round-trip for all three new types.

Verification:
  - go vet ./... clean; go build ./... clean
  - make build regenerates cleanly (no drift)
  - make test (-tags=assert), make ci (-tags='debug assert'), go test -race all pass
Close another scalar data-type gap. The library now supports 27 Arrow data
types: added MonthDayNanoInterval (arrow.MonthDayNanoInterval{Months, Days,
Nanoseconds}), the third interval type alongside DayTimeInterval and
MonthInterval.

tmpldata (pkg/object/objects.tmpldata):
  - MonthDayNanoInterval entry mirroring DayTimeInterval's structure. Comparison
    uses arrow.MonthDayNanoInterval.Cmp() for correct lexicographic ordering of
    (Months, Days, Nanoseconds) rather than error-prone manual field logic.
  - CastTo wires From {arrow.MonthDayNanoInterval, MonthDayNanoInterval
    (identity), Boolean}; all other conversions NotImplemented.
  - Cross-type From:MonthDayNanoInterval entries added to all types: Boolean
    references the correct fields (Months/Days/Nanoseconds), String uses %#v,
    the rest NotImplemented. The DayTimeInterval <-> MonthDayNanoInterval
    cross-conversion is NotImplemented because the structs are not directly
    convertible.

iterator (pkg/iterator/asjson.go):
  - monthDayNanoIntervalAsJSON mirroring dayTimeIntervalAsJSON.

Tests (pkg/object/monthdaynanointerval_test.go):
  - Construction, lexicographic comparison (including day-field tiebreak), and
    casts from the native type and Boolean.

Verification:
  - go vet ./... clean; go build ./... clean
  - make build regenerates cleanly (no drift)
  - make test, make ci, go test -race all pass
…mal256 cast/JSON, binary base64)

Addresses findings from the data-type review.

BLOCKER — CastElement (pkg/dataframe/element.go):
  The hand-written CastElement dispatch only covered the original numeric/date
  types, so DataFrame equality/comparison operations panicked for every other
  type — including all 5 newly-added types (Decimal256, Binary, FixedSizeBinary,
  LargeString, MonthDayNanoInterval) plus pre-existing gaps (Boolean, Decimal128,
  String, Timestamp, Duration, Float16, Time32/64, MonthInterval,
  DayTimeInterval). Added dispatch cases for every type that has a generated
  Element implementation. Added TestCastElementCoversAllTypes as a regression
  test covering all 16 reachable types.

MAJOR — CastToDecimal256 native type (pkg/object/objects.tmpldata):
  Decimal256's CastTo was missing a From:decimal256.Num entry, so
  CastToDecimal256(decimal256.Num) returned false and SmartBuilder rejected
  native Arrow decimal256 values. Added the native identity cast. Added
  TestCastToDecimal256Native.

MAJOR — Decimal256 UnmarshalJSON null panic (pkg/object/decimal.go):
  Unmarshaling JSON null into a Decimal256 nil-dereferenced because aux was a
  pointer. Now unmarshals into a struct value so null decodes to the zero
  Decimal256 without panicking. Added TestDecimal256JSONNull.

MAJOR — Binary/FixedSizeBinary JSON serialization (pkg/iterator/asjson.go):
  binaryAsJSON/fixedSizeBinaryAsJSON returned string(v.([]byte)), which silently
  corrupts non-UTF-8 bytes during JSON encoding. Now returns the raw []byte so
  encoding/json base64-encodes it, matching Apache Arrow's binary JSON
  serialization (lossless round-trip).

Verification:
  - go vet ./... clean; go build ./... clean
  - make test, make ci, go test -race all pass
Add the Arrow 14+ view types, bringing the total to 29 supported data types.

tmpldata (pkg/object/objects.tmpldata):
  - StringView: Type string, mirrors String/LargeString. name="string_view"
    to match arrow.StringViewType.Name().
  - BinaryView: Type []byte, mirrors Binary. name="binary_view".
    Lexicographic comparison via bytes.Compare/bytes.Equal.
  - Both ExcludeGenerate valueiterator/chunkiterator (array.StringView/
    BinaryView expose Value(i) but no bulk Values() accessor).

Hand-written value iterators (pkg/iterator/):
  - stringviewvalueiterator.go, binaryviewvalueiterator.go — mirror the
    String/Binary iterators against the generic ChunkIterator.
  - stringViewAsJSON / binaryViewAsJSON in asjson.go (binary returns raw
    []byte for base64 JSON encoding).

CastElement (pkg/dataframe/element.go):
  - Added *arrow.StringViewType and *arrow.BinaryViewType dispatch cases so
    DataFrame equality/comparison operations don't panic for view columns.
  - Extended TestCastElementCoversAllTypes to cover both view types.

Verification:
  - go vet ./... clean; go build ./... clean
  - make build regenerates cleanly (no drift)
  - make test, make ci, go test -race all pass
…rcleCI

The existing GitHub Actions workflow pinned go-version: '1.20', which cannot
build the module after the upgrade (go.mod now requires go 1.25). The check run
on PR #2 failed in ~16s for exactly this reason. CircleCI (.circleci/config.yml)
is long-obsolete (golang:1.14.4) and superseded by GitHub Actions.

Changes:
  - .github/workflows/go.yml: use actions/setup-go@v5 with go-version-file:
    go.mod so CI always tracks the module's declared Go version (currently
    1.25.0); install goimports (required by make build's codegen to format
    generated .gen.go files); run make build, make test, and make ci in
    distinct steps mirroring the local developer flow. Enable module cache.
  - Remove .circleci/ entirely (obsolete, references Go 1.14.4).

Verified locally: `make clean && make build` regenerates all codegen from
scratch with no drift, and `make test` / `make ci` both pass.
The previous commit removed the obsolete CircleCI config but the workflow
modernization didn't get staged with it. This completes the CI update:
actions/setup-go@v5 with go-version-file: go.mod (so CI always tracks the
module's Go version, currently 1.25.0) plus an explicit goimports install
(required by make build's codegen) and make build / make test / make ci steps.
@nickpoorman
nickpoorman merged commit c2ab629 into master Jul 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant