Skip to content

[C-API] Initial support for accessing file system - #19086

Merged
hannes merged 9 commits into
duckdb:mainfrom
Maxxen:c-api-filesystem
Sep 24, 2025
Merged

hannes merged 9 commits into
duckdb:mainfrom
Maxxen:c-api-filesystem

Conversation

@Maxxen

@Maxxen Maxxen commented Sep 22, 2025

Copy link
Copy Markdown
Member

This PR adds initial support for accessing DuckDB's filesystem layer from the C-API.

With the current state of the C-API this is primarily useful when writing table functions that scan some external file format, as it allows you to make use of DuckDBs existing cross platform filesystem, as well as any loaded extension file systems (like httpfs), without having to reimplement the equivalent functionality yourself.

I've tried to keep this initial version of the interface as small and simple as possible, while also leaving room to expand it in the future. The most complex part is the error handling, but I think I've solved it relatively neatly by having both the duckdb_file_system and duckdb_file_handle store their last occurred error internally, which can then be converted to duckdb_error_data through a pair of separate functions. Other than that the read/write functions follow posix semantics, returning the number of bytes read/written, 0 on EOF and a negative value on error. In practice the error value is always -1, but I've left it unspecified so that we can potentially return specific error codes in the future (like something like EAGAIN, when we get asyncio), without forcing the user to allocate, inspect and destroy a duckdb_error_data object to handle "common" IO errors.

@Maxxen Maxxen changed the title Initial support for accessing file system through C-API [C-API] Initial support for accessing file system Sep 22, 2025
Comment thread src/include/duckdb/main/capi/header_generation/header_base.hpp.template Outdated
Comment thread src/include/duckdb/main/capi/header_generation/header_base.hpp.template Outdated
Comment thread src/include/duckdb/main/capi/header_generation/header_base.hpp.template Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/main/capi/file_system-c.cpp
Comment thread test/api/capi/capi_file_system.cpp
Comment thread test/api/capi/capi_file_system.cpp Outdated
Comment thread test/api/capi/capi_file_system.cpp
Comment thread test/api/capi/capi_file_system.cpp
@duckdb-draftbot
duckdb-draftbot marked this pull request as draft September 23, 2025 09:39
@Maxxen

Maxxen commented Sep 23, 2025

Copy link
Copy Markdown
Member Author

Since we always have a filesystem in the client context (even if all are disabled, we still have a VirtualFileSystem), I've removed the duckdb_connection_get_file_system (redundant), and made duckdb_client_context_get_file_system infallible instead.

@Maxxen
Maxxen marked this pull request as ready for review September 23, 2025 10:24
@duckdb-draftbot
duckdb-draftbot marked this pull request as draft September 23, 2025 11:29
@Maxxen
Maxxen marked this pull request as ready for review September 23, 2025 12:50

@taniabogatsch taniabogatsch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! Left more comments, mostly nits. :)

Comment thread src/include/duckdb.h
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/include/duckdb.h Outdated
Comment thread src/main/capi/file_system-c.cpp
Comment thread src/main/capi/file_system-c.cpp
Comment thread test/api/capi/capi_file_system.cpp Outdated
Comment thread src/include/duckdb/main/capi/header_generation/header_base.hpp.template Outdated
@duckdb-draftbot
duckdb-draftbot marked this pull request as draft September 23, 2025 20:49
@Maxxen
Maxxen marked this pull request as ready for review September 23, 2025 20:50
@hannes
hannes merged commit 8d72b0e into duckdb:main Sep 24, 2025
96 checks passed
@hannes

hannes commented Sep 24, 2025

Copy link
Copy Markdown
Member

thanks!

krlmlr added a commit to krlmlr/duckdb-r-old that referenced this pull request Oct 21, 2025
[C-API] Initial support for accessing file system (duckdb/duckdb#19086)
Skip serialization of row_start in DataPointer when targeting latest storage (duckdb/duckdb#19111)
Allow directory override for extension build (duckdb/duckdb#19110)
krlmlr added a commit to krlmlr/duckdb-r-old that referenced this pull request Nov 1, 2025
[C-API] Initial support for accessing file system (duckdb/duckdb#19086)
Skip serialization of row_start in DataPointer when targeting latest storage (duckdb/duckdb#19111)
Allow directory override for extension build (duckdb/duckdb#19110)
krlmlr added a commit to krlmlr/duckdb-r-old that referenced this pull request Nov 2, 2025
[C-API] Initial support for accessing file system (duckdb/duckdb#19086)
Skip serialization of row_start in DataPointer when targeting latest storage (duckdb/duckdb#19111)
Allow directory override for extension build (duckdb/duckdb#19110)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants