Skip to content

Conversation

@universalmind303
Copy link
Contributor

@universalmind303 universalmind303 commented Dec 18, 2025

Changes Made

adds the following methods to most array implementations

// convert to type erased ArrayRef
// all arrays implement this
fn to_arrow(&self) -> DaftResult<ArrayRef>;
// convert to a concrete type which varies for each array type
// such as DataArray<Int8Type> -> arrow::array::Int8Array
// most arrays implement this, but not all of them. 
// this is also true for the original `as_arrow` (now `as_arrow2`) that this is intended to replace.
fn as_arrow(&self) -> DaftResult<Self::ArrowOutput>

and from methods.

// this is also implemented for all arrays. 
fn from_arrow<F: Into<Arc<Field>>>(field: F, array: ArrayRef) -> DaftResult<Self>;

Note for reviewers

the bulk of the changes is in the following

  • src/daft-core/src/array/ops/from_arrow.rs
  • src/daft-core/src/datatypes/logical.rs
  • src/daft-core/src/array/*

Related Issues

@universalmind303 universalmind303 changed the base branch from main to universalmind303/dtype_and_field_conversions December 18, 2025 19:24
Base automatically changed from universalmind303/dtype_and_field_conversions to main December 18, 2025 20:06
@universalmind303 universalmind303 changed the title wip: array & series to/from arrow refactor(arrow2): array & series to/from arrow Dec 18, 2025
@codecov
Copy link

codecov bot commented Dec 18, 2025

Codecov Report

❌ Patch coverage is 74.52153% with 213 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.33%. Comparing base (1b2e5cd) to head (84073bb).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/daft-core/src/array/ops/from_arrow.rs 71.46% 125 Missing ⚠️
src/daft-core/src/datatypes/logical.rs 38.98% 36 Missing ⚠️
src/daft-core/src/array/ops/as_arrow.rs 88.81% 18 Missing ⚠️
src/daft-core/src/datatypes/python.rs 21.42% 11 Missing ⚠️
src/daft-functions-binary/src/kernels.rs 37.50% 5 Missing ⚠️
...c/daft-core/src/array/ops/approx_count_distinct.rs 0.00% 4 Missing ⚠️
src/daft-core/src/array/struct_array.rs 78.94% 4 Missing ⚠️
...c/daft-core/src/series/array_impl/logical_array.rs 50.00% 4 Missing ⚠️
...rc/daft-core/src/series/array_impl/python_array.rs 60.00% 2 Missing ⚠️
src/daft-core/src/array/fixed_size_list_array.rs 97.14% 1 Missing ⚠️
... and 3 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5848      +/-   ##
==========================================
+ Coverage   71.41%   72.33%   +0.91%     
==========================================
  Files         964      964              
  Lines      127899   125650    -2249     
==========================================
- Hits        91345    90889     -456     
+ Misses      36554    34761    -1793     
Files with missing lines Coverage Δ
src/daft-core/src/array/file_array.rs 61.32% <100.00%> (ø)
src/daft-core/src/array/from.rs 69.86% <100.00%> (+1.74%) ⬆️
src/daft-core/src/array/growable/arrow_growable.rs 100.00% <100.00%> (ø)
src/daft-core/src/array/list_array.rs 85.34% <100.00%> (+1.24%) ⬆️
src/daft-core/src/array/ops/approx_sketch.rs 100.00% <100.00%> (ø)
src/daft-core/src/array/ops/arithmetic.rs 90.27% <ø> (ø)
src/daft-core/src/array/ops/arrow/comparison.rs 99.10% <ø> (ø)
src/daft-core/src/array/ops/cast.rs 81.42% <100.00%> (ø)
src/daft-core/src/array/ops/compare_agg.rs 65.94% <ø> (ø)
src/daft-core/src/array/ops/comparison.rs 79.85% <100.00%> (ø)
... and 35 more

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@universalmind303 universalmind303 marked this pull request as ready for review December 19, 2025 17:12
@universalmind303 universalmind303 merged commit 38cc5d5 into main Dec 19, 2025
70 of 72 checks passed
@universalmind303 universalmind303 deleted the universalmind303/series_to_arrow branch December 19, 2025 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants