-
Notifications
You must be signed in to change notification settings - Fork 81
Pass extended file info to parquet scan & extended file info to Avro Scan to avoid unnecessary HEAD requests #268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pass extended file info to parquet scan & extended file info to Avro Scan to avoid unnecessary HEAD requests #268
Conversation
Tishj
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, great work 👍
| "kind": "git", | ||
| "repository": "https://github.com/duckdb/vcpkg-duckdb-ports", | ||
| "baseline": "0f9bf648ba1ee29291890a1ca9a49a80bba017eb", | ||
| "baseline": "d3a5de6bd45b5907ed189054dd23eff6ec869e14", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is reverting the registry version:
https://github.com/duckdb/vcpkg-duckdb-ports/commits/main/
0f9bf648ba1ee29291890a1ca9a49a80bba017eb - duckdb/vcpkg-duckdb-ports@0f9bf64
d3aa25a569e0eb35e01c59e3e62c9bc7406f0ed9 - duckdb/vcpkg-duckdb-ports@d3aa25a
This version is older than the one that's currently there, is there a reason for that?
I plan to bump this to add avro writes, I'd like to not run into any unrelated issues due to this PR causing issues with a newer patch version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really see why this change is here, but I also don't think it matters, so we can merge it regardless
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new commit is the most recent commit to vcpkg-duckdb-ports
d3a5de6bd45b5907ed189054dd23eff6ec869e14 I don't know where d3aa25 is coming from
requires duckdb/duckdb-avro#28
requires duckdb/duckdb#17698
By passing an extended open file info to the avro scan, we can pass
validate_external_file_cache = falseto the FileReader, which means we won't make unnecessary HEAD requests on secondary reads.