Repository navigation
[unsupervised AI] Keep non-string object columns unchanged in from_pandas - #12626
Draft
barlowa124 wants to merge 3 commits into
Draft
barlowa124 wants to merge 3 commits into
barlowa124 wants to merge 3 commits into
Conversation
to_pyarrow_string converts every object-dtype column to string[pyarrow], which silently stringifies mixed columns like ["1", 1] (dask#12612). Since FromPandas holds the whole frame in memory, decide convertibility there once per column and index with a values check, so mixed data keeps its dtype and every partition agrees with the meta.
Object-dtype columns receive placeholder values in _meta_nonempty that\npyarrow cannot type-infer. Fall back to dtype-driven inference on the\nempty meta, which types them as null fields (explicit schema overrides\nstill apply). Also update the to_numeric expectation for a mixed\ncolumn that now stays object dtype.
Contributor
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 25 files ± 0 25 suites ±0 6h 15m 47s ⏱️ - 35m 47s Results for commit 0d00c92. ± Comparison against base commit 6ce3740. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
This PR was written autonomously by an AI agent and has not been reviewed by a human yet. Maintainers should ignore it until the human author has reviewed and understood everything that the AI agent wrote and approved it.
pixi run lintto_pyarrow_stringconverts every object-dtype column tostring[pyarrow], yet object dtype holds arbitrary Python objects. Amixed column such as
["1", 1]loses the1to stringification, soisinresults ondd.from_pandasdiffer from pandas in bothdirections.
FromPandasholds the whole frame in memory, so conversion now runsthere once, gated by a values check (
infer_dtypeon each column andindex level). Object columns whose non-missing values are not all
strings keep their dtype. Per-partition conversion is dropped: slices
of the converted frame already agree with the meta, and a per-slice
check could make partitions disagree.
The other
to_pyarrow_stringcallers (ArrowStringConversion, theread_sqlchunk path,assert_eq) are unchanged, since each sees asingle partition and cannot make a whole-column decision.
Verification was local, without pixi.
ruff checkon the touchedfiles reports only pre-existing findings on unmodified lines, and the
new regression test plus the dataframe/pyarrow and
dask_exprcollection/core suites all pass.