Skip to content
This repository was archived by the owner on Oct 10, 2025. It is now read-only.

Implement unity catalog extension - #4890

Merged
acquamarin merged 6 commits into
masterfrom
new-unity-catalog
Feb 14, 2025
Merged

Implement unity catalog extension#4890
acquamarin merged 6 commits into
masterfrom
new-unity-catalog

Conversation

@acquamarin

@acquamarin acquamarin commented Feb 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR allows kuzu to scan from unity catalog.
To scan from unity catalog, users have to attach to the unity catalog first:

attach "catalog_name" as alias (dbtype uc_catalog)

Then users can query unity catalog tables as other relational tables.

LOAD FROM unity.person RETURN *;

The duckdb's implementation of unity catalog is buggy right now, so our unity-catalog extension is also an experimental feature.

@github-actions

github-actions Bot commented Feb 12, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: dfabf90eab17ec0dc0f87d18464152412e1fd8ee
Branch commit hash: 3d6321c332746c7c34bd1bcdd5053e75ffa3fdfa

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 723.00 736.85 -13.86 (-1.88%)
aggregation q28 6347.76 6358.04 -10.28 (-0.16%)
filter q14 126.44 128.29 -1.85 (-1.44%)
filter q15 127.12 126.50 0.61 (0.49%)
filter q16 305.19 306.18 -1.00 (-0.33%)
filter q17 453.57 446.79 6.77 (1.52%)
filter q18 1906.22 1922.93 -16.71 (-0.87%)
filter zonemap-node 88.92 88.87 0.05 (0.06%)
filter zonemap-node-lhs-cast 89.04 90.75 -1.71 (-1.88%)
filter zonemap-node-null 88.81 90.66 -1.85 (-2.04%)
filter zonemap-rel 5407.24 5394.15 13.09 (0.24%)
fixed_size_expr_evaluator q07 571.92 581.95 -10.03 (-1.72%)
fixed_size_expr_evaluator q08 801.39 801.57 -0.18 (-0.02%)
fixed_size_expr_evaluator q09 806.24 803.44 2.80 (0.35%)
fixed_size_expr_evaluator q10 241.56 236.67 4.89 (2.07%)
fixed_size_expr_evaluator q11 234.13 229.64 4.49 (1.96%)
fixed_size_expr_evaluator q12 226.42 231.70 -5.28 (-2.28%)
fixed_size_expr_evaluator q13 1462.73 1465.25 -2.51 (-0.17%)
fixed_size_seq_scan q23 110.59 111.76 -1.17 (-1.05%)
join q29 763.22 703.37 59.85 (8.51%)
join q30 10961.02 11083.57 -122.55 (-1.11%)
join q31 8.88 9.98 -1.10 (-11.06%)
join SelectiveTwoHopJoin 52.69 59.99 -7.30 (-12.17%)
ldbc_snb_ic q35 2615.92 2607.02 8.91 (0.34%)
ldbc_snb_ic q36 460.99 485.56 -24.56 (-5.06%)
ldbc_snb_is q32 4.40 4.47 -0.07 (-1.66%)
ldbc_snb_is q33 15.47 14.83 0.64 (4.34%)
ldbc_snb_is q34 1.18 1.25 -0.06 (-5.15%)
multi-rel multi-rel-large-scan 1381.52 1392.59 -11.07 (-0.79%)
multi-rel multi-rel-lookup 27.47 32.54 -5.07 (-15.59%)
multi-rel multi-rel-small-scan 76.76 102.16 -25.40 (-24.86%)
order_by q25 134.85 131.92 2.93 (2.22%)
order_by q26 451.05 452.45 -1.40 (-0.31%)
order_by q27 1471.34 1420.37 50.97 (3.59%)
recursive_join recursive-join-bidirection 306.12 296.22 9.90 (3.34%)
recursive_join recursive-join-dense 7405.06 7444.01 -38.95 (-0.52%)
recursive_join recursive-join-path 24360.77 24117.33 243.44 (1.01%)
recursive_join recursive-join-sparse 1050.55 1057.45 -6.90 (-0.65%)
recursive_join recursive-join-trail 7388.10 7418.08 -29.98 (-0.40%)
scan_after_filter q01 177.15 175.01 2.13 (1.22%)
scan_after_filter q02 164.66 159.85 4.81 (3.01%)
shortest_path_ldbc100 q37 96.21 97.65 -1.44 (-1.48%)
shortest_path_ldbc100 q38 339.35 377.28 -37.93 (-10.05%)
shortest_path_ldbc100 q39 45.93 64.85 -18.91 (-29.17%)
shortest_path_ldbc100 q40 451.98 464.15 -12.17 (-2.62%)
var_size_expr_evaluator q03 2154.04 2149.45 4.60 (0.21%)
var_size_expr_evaluator q04 2219.13 2203.44 15.68 (0.71%)
var_size_expr_evaluator q05 2611.88 2620.11 -8.23 (-0.31%)
var_size_expr_evaluator q06 1336.54 1345.39 -8.85 (-0.66%)
var_size_seq_scan q19 1468.05 1459.82 8.23 (0.56%)
var_size_seq_scan q20 2337.19 2352.12 -14.93 (-0.63%)
var_size_seq_scan q21 2310.06 2311.06 -1.01 (-0.04%)
var_size_seq_scan q22 128.59 128.13 0.46 (0.36%)

@codecov

codecov Bot commented Feb 12, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.49%. Comparing base (dfabf90) to head (6cd8c1e).
Report is 9 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4890      +/-   ##
==========================================
- Coverage   86.52%   86.49%   -0.03%     
==========================================
  Files        1403     1403              
  Lines       60519    60629     +110     
  Branches     7433     7458      +25     
==========================================
+ Hits        52364    52444      +80     
- Misses       7986     8016      +30     
  Partials      169      169

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@acquamarin
acquamarin requested a review from ray6080 February 13, 2025 04:18
ray6080
ray6080 approved these changes Feb 13, 2025
View reviewed changes
Comment thread Makefile
Comment thread extension/unity-catalog/src/connector/unity_catalog_connector.cpp
Comment thread extension/unity-catalog/test/test_files/unity_catalog.test
@github-actions

github-actions Bot commented Feb 13, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: b3f88d7439cf955555c297f70e0eb43c64d73d20
Branch commit hash: a5c84348ebd1d169e5084ebae1846b9eafadaca8

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 727.93 720.62 7.32 (1.02%)
aggregation q28 6342.55 6326.17 16.38 (0.26%)
filter q14 130.57 117.76 12.81 (10.88%)
filter q15 125.41 113.50 11.91 (10.50%)
filter q16 311.21 305.32 5.89 (1.93%)
filter q17 444.56 437.24 7.32 (1.67%)
filter q18 1917.24 1930.32 -13.08 (-0.68%)
filter zonemap-node 90.57 80.24 10.32 (12.87%)
filter zonemap-node-lhs-cast 92.60 80.42 12.18 (15.15%)
filter zonemap-node-null 89.40 80.69 8.71 (10.80%)
filter zonemap-rel 5506.43 5474.96 31.48 (0.57%)
fixed_size_expr_evaluator q07 588.58 568.88 19.70 (3.46%)
fixed_size_expr_evaluator q08 817.79 796.51 21.28 (2.67%)
fixed_size_expr_evaluator q09 817.19 793.48 23.72 (2.99%)
fixed_size_expr_evaluator q10 247.67 229.77 17.90 (7.79%)
fixed_size_expr_evaluator q11 240.72 222.30 18.42 (8.29%)
fixed_size_expr_evaluator q12 237.60 218.88 18.73 (8.56%)
fixed_size_expr_evaluator q13 1474.56 1448.38 26.18 (1.81%)
fixed_size_seq_scan q23 119.11 107.58 11.53 (10.72%)
join q29 741.08 717.94 23.14 (3.22%)
join q30 10321.31 9456.93 864.38 (9.14%)
join q31 5.01 5.94 -0.93 (-15.73%)
join SelectiveTwoHopJoin 55.54 54.19 1.35 (2.50%)
ldbc_snb_ic q35 2557.44 2601.03 -43.59 (-1.68%)
ldbc_snb_ic q36 478.58 459.63 18.95 (4.12%)
ldbc_snb_is q32 7.06 6.02 1.04 (17.33%)
ldbc_snb_is q33 12.52 15.48 -2.97 (-19.17%)
ldbc_snb_is q34 1.24 1.22 0.02 (1.56%)
multi-rel multi-rel-large-scan 1297.14 1406.99 -109.85 (-7.81%)
multi-rel multi-rel-lookup 31.33 5.52 25.82 (467.90%)
multi-rel multi-rel-small-scan 84.52 63.38 21.15 (33.37%)
order_by q25 130.85 119.85 11.00 (9.18%)
order_by q26 461.75 440.72 21.03 (4.77%)
order_by q27 1441.00 1439.68 1.32 (0.09%)
recursive_join recursive-join-bidirection 300.66 311.42 -10.75 (-3.45%)
recursive_join recursive-join-dense 7353.75 7376.92 -23.17 (-0.31%)
recursive_join recursive-join-path 23874.97 23650.29 224.68 (0.95%)
recursive_join recursive-join-sparse 1066.37 1059.85 6.52 (0.62%)
recursive_join recursive-join-trail 7345.56 7316.55 29.01 (0.40%)
scan_after_filter q01 174.88 170.50 4.38 (2.57%)
scan_after_filter q02 161.14 155.04 6.11 (3.94%)
shortest_path_ldbc100 q37 91.54 95.66 -4.12 (-4.30%)
shortest_path_ldbc100 q38 381.96 394.82 -12.87 (-3.26%)
shortest_path_ldbc100 q39 63.71 65.30 -1.59 (-2.44%)
shortest_path_ldbc100 q40 448.99 444.73 4.25 (0.96%)
var_size_expr_evaluator q03 2114.72 2069.06 45.66 (2.21%)
var_size_expr_evaluator q04 2275.87 2243.26 32.61 (1.45%)
var_size_expr_evaluator q05 2963.38 2667.22 296.16 (11.10%)
var_size_expr_evaluator q06 1349.17 1324.83 24.33 (1.84%)
var_size_seq_scan q19 1466.33 1443.40 22.93 (1.59%)
var_size_seq_scan q20 2359.36 2379.78 -20.42 (-0.86%)
var_size_seq_scan q21 2268.78 2307.51 -38.74 (-1.68%)
var_size_seq_scan q22 127.69 123.92 3.77 (3.04%)

@acquamarin acquamarin mentioned this pull request Feb 13, 2025
Closed
26 tasks
@github-actions

github-actions Bot commented Feb 13, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: b93875e2210bf45cd8aa45e6c1516ed6472b10e9
Branch commit hash: 70b1dbb35f5aad8cbbdcadfd34da69d4fef5c00a

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 728.36 722.67 5.69 (0.79%)
aggregation q28 6559.37 6358.22 201.14 (3.16%)
filter q14 126.20 126.27 -0.07 (-0.06%)
filter q15 124.14 129.03 -4.90 (-3.80%)
filter q16 306.40 306.21 0.19 (0.06%)
filter q17 450.75 446.33 4.42 (0.99%)
filter q18 1986.31 2020.28 -33.96 (-1.68%)
filter zonemap-node 90.10 89.17 0.93 (1.04%)
filter zonemap-node-lhs-cast 88.67 89.30 -0.63 (-0.70%)
filter zonemap-node-null 88.35 88.96 -0.60 (-0.68%)
filter zonemap-rel 5429.31 5445.48 -16.17 (-0.30%)
fixed_size_expr_evaluator q07 571.11 569.55 1.55 (0.27%)
fixed_size_expr_evaluator q08 801.37 802.39 -1.02 (-0.13%)
fixed_size_expr_evaluator q09 799.66 803.05 -3.39 (-0.42%)
fixed_size_expr_evaluator q10 236.08 238.58 -2.50 (-1.05%)
fixed_size_expr_evaluator q11 229.53 228.87 0.66 (0.29%)
fixed_size_expr_evaluator q12 226.27 226.20 0.07 (0.03%)
fixed_size_expr_evaluator q13 1455.24 1458.75 -3.51 (-0.24%)
fixed_size_seq_scan q23 109.35 108.88 0.47 (0.43%)
join q29 730.58 719.99 10.59 (1.47%)
join q30 11222.24 10608.40 613.83 (5.79%)
join q31 5.36 4.56 0.80 (17.60%)
join SelectiveTwoHopJoin 57.40 55.62 1.78 (3.19%)
ldbc_snb_ic q35 2603.71 2618.21 -14.49 (-0.55%)
ldbc_snb_ic q36 446.14 448.47 -2.33 (-0.52%)
ldbc_snb_is q32 3.94 6.44 -2.49 (-38.74%)
ldbc_snb_is q33 13.73 14.33 -0.60 (-4.19%)
ldbc_snb_is q34 1.19 1.34 -0.15 (-11.06%)
multi-rel multi-rel-large-scan 1359.62 1703.53 -343.91 (-20.19%)
multi-rel multi-rel-lookup 10.97 33.36 -22.39 (-67.10%)
multi-rel multi-rel-small-scan 51.39 99.94 -48.54 (-48.57%)
order_by q25 127.34 135.48 -8.15 (-6.01%)
order_by q26 463.50 465.23 -1.73 (-0.37%)
order_by q27 1422.26 1440.76 -18.51 (-1.28%)
recursive_join recursive-join-bidirection 323.18 295.97 27.21 (9.19%)
recursive_join recursive-join-dense 6802.33 7352.70 -550.37 (-7.49%)
recursive_join recursive-join-path 24144.01 24351.28 -207.27 (-0.85%)
recursive_join recursive-join-sparse 1054.42 1057.65 -3.23 (-0.30%)
recursive_join recursive-join-trail 7071.50 7379.07 -307.57 (-4.17%)
scan_after_filter q01 171.88 172.01 -0.13 (-0.08%)
scan_after_filter q02 159.96 157.25 2.71 (1.72%)
shortest_path_ldbc100 q37 92.60 84.98 7.62 (8.96%)
shortest_path_ldbc100 q38 385.37 371.14 14.23 (3.83%)
shortest_path_ldbc100 q39 59.92 62.25 -2.33 (-3.74%)
shortest_path_ldbc100 q40 466.18 454.36 11.82 (2.60%)
var_size_expr_evaluator q03 2127.73 2115.22 12.51 (0.59%)
var_size_expr_evaluator q04 2313.57 2357.49 -43.92 (-1.86%)
var_size_expr_evaluator q05 2642.46 2655.44 -12.97 (-0.49%)
var_size_expr_evaluator q06 1380.42 1383.24 -2.82 (-0.20%)
var_size_seq_scan q19 1485.27 1488.83 -3.56 (-0.24%)
var_size_seq_scan q20 2321.74 2312.22 9.52 (0.41%)
var_size_seq_scan q21 2312.42 2289.68 22.73 (0.99%)
var_size_seq_scan q22 128.95 128.27 0.68 (0.53%)

@github-actions

github-actions Bot commented Feb 14, 2025

Copy link
Copy Markdown

Benchmark Result

Master commit hash: a88d57e3d3a1323feb88a31f946171cd17f20ac9
Branch commit hash: eea02f81c9847b20711d30ae007e5c5b48b23ea5

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 723.60 724.38 -0.78 (-0.11%)
aggregation q28 6358.72 6365.18 -6.45 (-0.10%)
filter q14 126.07 126.15 -0.07 (-0.06%)
filter q15 127.94 125.06 2.88 (2.31%)
filter q16 314.87 301.31 13.56 (4.50%)
filter q17 447.40 446.32 1.09 (0.24%)
filter q18 1894.21 1939.86 -45.65 (-2.35%)
filter zonemap-node 88.53 90.69 -2.16 (-2.38%)
filter zonemap-node-lhs-cast 88.67 88.98 -0.31 (-0.35%)
filter zonemap-node-null 89.98 88.59 1.38 (1.56%)
filter zonemap-rel 5449.51 5422.24 27.27 (0.50%)
fixed_size_expr_evaluator q07 582.52 578.36 4.17 (0.72%)
fixed_size_expr_evaluator q08 809.12 807.96 1.16 (0.14%)
fixed_size_expr_evaluator q09 809.40 811.59 -2.20 (-0.27%)
fixed_size_expr_evaluator q10 247.33 245.64 1.68 (0.69%)
fixed_size_expr_evaluator q11 238.68 238.22 0.45 (0.19%)
fixed_size_expr_evaluator q12 235.86 236.79 -0.92 (-0.39%)
fixed_size_expr_evaluator q13 1464.94 1456.47 8.47 (0.58%)
fixed_size_seq_scan q23 117.85 117.14 0.71 (0.60%)
join q29 765.61 732.30 33.31 (4.55%)
join q30 9872.52 10505.58 -633.06 (-6.03%)
join q31 9.32 7.93 1.39 (17.52%)
join SelectiveTwoHopJoin 56.21 57.26 -1.05 (-1.84%)
ldbc_snb_ic q35 2585.91 2603.11 -17.20 (-0.66%)
ldbc_snb_ic q36 487.39 482.67 4.72 (0.98%)
ldbc_snb_is q32 6.34 5.32 1.02 (19.23%)
ldbc_snb_is q33 15.30 17.42 -2.12 (-12.15%)
ldbc_snb_is q34 1.14 1.10 0.04 (3.91%)
multi-rel multi-rel-large-scan 1795.72 1642.31 153.41 (9.34%)
multi-rel multi-rel-lookup 22.65 21.48 1.17 (5.44%)
multi-rel multi-rel-small-scan 83.71 92.81 -9.10 (-9.80%)
order_by q25 133.69 131.07 2.62 (2.00%)
order_by q26 458.63 454.88 3.75 (0.82%)
order_by q27 1404.31 1406.59 -2.27 (-0.16%)
recursive_join recursive-join-bidirection 286.69 310.91 -24.22 (-7.79%)
recursive_join recursive-join-dense 7378.07 7376.56 1.51 (0.02%)
recursive_join recursive-join-path 24064.80 24322.43 -257.63 (-1.06%)
recursive_join recursive-join-sparse 1059.86 1047.53 12.33 (1.18%)
recursive_join recursive-join-trail 7358.34 7353.06 5.28 (0.07%)
scan_after_filter q01 172.53 170.35 2.19 (1.28%)
scan_after_filter q02 156.79 160.39 -3.60 (-2.25%)
shortest_path_ldbc100 q37 82.26 88.63 -6.37 (-7.19%)
shortest_path_ldbc100 q38 377.30 371.13 6.17 (1.66%)
shortest_path_ldbc100 q39 64.53 66.35 -1.82 (-2.74%)
shortest_path_ldbc100 q40 462.24 426.33 35.91 (8.42%)
var_size_expr_evaluator q03 2081.20 2082.96 -1.76 (-0.08%)
var_size_expr_evaluator q04 2204.27 2225.81 -21.55 (-0.97%)
var_size_expr_evaluator q05 2924.58 2628.58 296.00 (11.26%)
var_size_expr_evaluator q06 1325.89 1334.23 -8.35 (-0.63%)
var_size_seq_scan q19 1462.41 1464.03 -1.62 (-0.11%)
var_size_seq_scan q20 2316.95 2346.12 -29.17 (-1.24%)
var_size_seq_scan q21 2271.35 2278.22 -6.86 (-0.30%)
var_size_seq_scan q22 127.03 127.74 -0.71 (-0.56%)

@acquamarin
acquamarin merged commit edf39b9 into master Feb 14, 2025
@acquamarin
acquamarin deleted the new-unity-catalog branch February 14, 2025 21:47
acquamarin added a commit that referenced this pull request Feb 19, 2025
@acquamarin
Implement unity catalog extension (#4890)
b3a61ea
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@ray6080 ray6080 ray6080 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@acquamarin @ray6080