Skip to content
This repository was archived by the owner on Oct 10, 2025. It is now read-only.

Fix scan pandas/polars type error - #5218

Merged
acquamarin merged 2 commits into
masterfrom
fix-test-scan-pandas-empty-list
Apr 10, 2025
Merged

Fix scan pandas/polars type error#5218
acquamarin merged 2 commits into
masterfrom
fix-test-scan-pandas-empty-list

Conversation

@acquamarin

@acquamarin acquamarin commented Apr 10, 2025

Copy link
Copy Markdown
Contributor

As desribed in #5216, our current pandas/polars scanner is not able to handle empty lists.
This PR correctly handles the empty list issue in polars/pandas.
Closes #5216

@acquamarin
acquamarin requested a review from andyfengHKU April 10, 2025 05:48
@codecov

codecov Bot commented Apr 10, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.98%. Comparing base (dbdc2f0) to head (3866da0).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #5218   +/-   ##
=======================================
  Coverage   86.97%   86.98%           
=======================================
  Files        1405     1405           
  Lines       61629    61629           
  Branches     7540     7541    +1     
=======================================
+ Hits        53603    53605    +2     
+ Misses       7853     7851    -2     
  Partials      173      173

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Apr 10, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: dbdc2f0939a8b6a6b9144e80e587d20c12d65b7d
Branch commit hash: 052f1b433014e2e4232c3342c423e6b629f9a9a8

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 747.19 740.26 6.94 (0.94%)
aggregation q28 6599.78 6706.97 -107.19 (-1.60%)
filter q14 142.44 144.26 -1.82 (-1.26%)
filter q15 142.45 141.14 1.30 (0.92%)
filter q16 359.10 359.64 -0.54 (-0.15%)
filter q17 463.39 463.64 -0.25 (-0.05%)
filter q18 1899.87 1895.78 4.09 (0.22%)
filter zonemap-node 96.97 97.37 -0.39 (-0.41%)
filter zonemap-node-lhs-cast 97.16 97.21 -0.05 (-0.05%)
filter zonemap-node-null 96.90 97.34 -0.44 (-0.45%)
filter zonemap-rel 5552.67 5547.24 5.43 (0.10%)
fixed_size_expr_evaluator q07 695.23 695.27 -0.05 (-0.01%)
fixed_size_expr_evaluator q08 978.46 974.69 3.78 (0.39%)
fixed_size_expr_evaluator q09 973.56 975.99 -2.43 (-0.25%)
fixed_size_expr_evaluator q10 270.35 270.97 -0.62 (-0.23%)
fixed_size_expr_evaluator q11 270.53 271.40 -0.87 (-0.32%)
fixed_size_expr_evaluator q12 249.28 249.25 0.03 (0.01%)
fixed_size_expr_evaluator q13 1579.05 1583.24 -4.19 (-0.26%)
fixed_size_seq_scan q23 131.92 125.75 6.16 (4.90%)
join q29 762.89 707.98 54.91 (7.76%)
join q30 1693.15 1513.12 180.03 (11.90%)
join q31 5.02 5.36 -0.33 (-6.25%)
join SelectiveTwoHopJoin 56.38 46.90 9.47 (20.19%)
ldbc_snb_ic q35 9.17 10.17 -1.00 (-9.84%)
ldbc_snb_ic q36 95.67 98.81 -3.14 (-3.18%)
ldbc_snb_is q32 6.05 3.73 2.32 (62.26%)
ldbc_snb_is q33 13.61 16.11 -2.50 (-15.52%)
ldbc_snb_is q34 1.18 1.13 0.05 (4.49%)
multi-rel multi-rel-large-scan 1701.08 1697.56 3.52 (0.21%)
multi-rel multi-rel-lookup 10.33 11.97 -1.63 (-13.63%)
multi-rel multi-rel-small-scan 199.97 206.92 -6.95 (-3.36%)
order_by q25 147.23 151.05 -3.81 (-2.53%)
order_by q26 464.37 468.27 -3.90 (-0.83%)
order_by q27 1389.64 1403.62 -13.98 (-1.00%)
recursive_join recursive-join-bidirection 298.60 297.74 0.86 (0.29%)
recursive_join recursive-join-dense 6994.94 7017.64 -22.70 (-0.32%)
recursive_join recursive-join-path 23354.03 23316.38 37.65 (0.16%)
recursive_join recursive-join-sparse 628.31 632.49 -4.18 (-0.66%)
recursive_join recursive-join-trail 7006.12 6957.02 49.10 (0.71%)
scan_after_filter q01 185.88 188.46 -2.58 (-1.37%)
scan_after_filter q02 172.78 173.80 -1.02 (-0.58%)
shortest_path_ldbc100 q37 96.52 99.07 -2.55 (-2.58%)
shortest_path_ldbc100 q38 258.55 344.52 -85.97 (-24.95%)
shortest_path_ldbc100 q39 56.50 65.73 -9.23 (-14.04%)
shortest_path_ldbc100 q40 334.86 368.46 -33.60 (-9.12%)
var_size_expr_evaluator q03 2119.11 2118.96 0.15 (0.01%)
var_size_expr_evaluator q04 2191.74 2215.26 -23.51 (-1.06%)
var_size_expr_evaluator q05 2614.43 2652.00 -37.57 (-1.42%)
var_size_expr_evaluator q06 1370.58 1387.46 -16.88 (-1.22%)
var_size_seq_scan q19 1438.83 1453.85 -15.02 (-1.03%)
var_size_seq_scan q20 2536.93 2569.27 -32.34 (-1.26%)
var_size_seq_scan q21 2265.34 2371.88 -106.54 (-4.49%)
var_size_seq_scan q22 128.29 131.28 -2.99 (-2.28%)

@acquamarin
acquamarin force-pushed the fix-test-scan-pandas-empty-list branch from ad4d741 to 3866da0 Compare April 10, 2025 06:53
@github-actions

github-actions Bot commented Apr 10, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: dbdc2f0939a8b6a6b9144e80e587d20c12d65b7d
Branch commit hash: 1a8f097a6cc1aa71ad54aa679aceaa191d9ec806

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 743.32 740.26 3.06 (0.41%)
aggregation q28 6578.44 6706.97 -128.52 (-1.92%)
filter q14 142.19 144.26 -2.07 (-1.44%)
filter q15 144.19 141.14 3.04 (2.16%)
filter q16 360.34 359.64 0.71 (0.20%)
filter q17 462.51 463.64 -1.13 (-0.24%)
filter q18 1890.66 1895.78 -5.13 (-0.27%)
filter zonemap-node 97.53 97.37 0.16 (0.17%)
filter zonemap-node-lhs-cast 97.75 97.21 0.55 (0.56%)
filter zonemap-node-null 97.42 97.34 0.08 (0.08%)
filter zonemap-rel 5596.03 5547.24 48.79 (0.88%)
fixed_size_expr_evaluator q07 697.15 695.27 1.88 (0.27%)
fixed_size_expr_evaluator q08 974.63 974.69 -0.06 (-0.01%)
fixed_size_expr_evaluator q09 973.44 975.99 -2.55 (-0.26%)
fixed_size_expr_evaluator q10 269.55 270.97 -1.42 (-0.53%)
fixed_size_expr_evaluator q11 269.87 271.40 -1.53 (-0.56%)
fixed_size_expr_evaluator q12 248.69 249.25 -0.56 (-0.22%)
fixed_size_expr_evaluator q13 1581.64 1583.24 -1.60 (-0.10%)
fixed_size_seq_scan q23 129.67 125.75 3.91 (3.11%)
join q29 757.09 707.98 49.11 (6.94%)
join q30 1734.71 1513.12 221.59 (14.64%)
join q31 5.04 5.36 -0.32 (-5.94%)
join SelectiveTwoHopJoin 56.46 46.90 9.56 (20.38%)
ldbc_snb_ic q35 10.06 10.17 -0.11 (-1.04%)
ldbc_snb_ic q36 101.87 98.81 3.05 (3.09%)
ldbc_snb_is q32 6.36 3.73 2.63 (70.41%)
ldbc_snb_is q33 12.73 16.11 -3.37 (-20.94%)
ldbc_snb_is q34 1.29 1.13 0.16 (13.96%)
multi-rel multi-rel-large-scan 1717.30 1697.56 19.74 (1.16%)
multi-rel multi-rel-lookup 10.07 11.97 -1.89 (-15.83%)
multi-rel multi-rel-small-scan 175.67 206.92 -31.26 (-15.11%)
order_by q25 143.72 151.05 -7.32 (-4.85%)
order_by q26 463.19 468.27 -5.08 (-1.08%)
order_by q27 1388.09 1403.62 -15.53 (-1.11%)
recursive_join recursive-join-bidirection 326.03 297.74 28.29 (9.50%)
recursive_join recursive-join-dense 7163.18 7017.64 145.54 (2.07%)
recursive_join recursive-join-path 23793.05 23316.38 476.67 (2.04%)
recursive_join recursive-join-sparse 633.54 632.49 1.05 (0.17%)
recursive_join recursive-join-trail 7090.25 6957.02 133.23 (1.91%)
scan_after_filter q01 188.82 188.46 0.36 (0.19%)
scan_after_filter q02 176.61 173.80 2.81 (1.62%)
shortest_path_ldbc100 q37 83.64 99.07 -15.43 (-15.57%)
shortest_path_ldbc100 q38 379.38 344.52 34.86 (10.12%)
shortest_path_ldbc100 q39 59.74 65.73 -5.98 (-9.11%)
shortest_path_ldbc100 q40 388.17 368.46 19.71 (5.35%)
var_size_expr_evaluator q03 2093.87 2118.96 -25.09 (-1.18%)
var_size_expr_evaluator q04 2185.16 2215.26 -30.09 (-1.36%)
var_size_expr_evaluator q05 2617.45 2652.00 -34.54 (-1.30%)
var_size_expr_evaluator q06 1355.37 1387.46 -32.09 (-2.31%)
var_size_seq_scan q19 1440.09 1453.85 -13.76 (-0.95%)
var_size_seq_scan q20 2474.40 2569.27 -94.87 (-3.69%)
var_size_seq_scan q21 2265.12 2371.88 -106.76 (-4.50%)
var_size_seq_scan q22 128.63 131.28 -2.65 (-2.02%)

@acquamarin
acquamarin merged commit bf30781 into master Apr 10, 2025
@acquamarin
acquamarin deleted the fix-test-scan-pandas-empty-list branch April 10, 2025 08:10
@prrao87 prrao87 changed the title Fix scan pandas type error Fix scan pandas/polars type error May 6, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@andyfengHKU andyfengHKU andyfengHKU approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug: JSON extension does not handle empty arrays

2 participants

@acquamarin @andyfengHKU