Skip to content

Connect passthrough - #25990

Open
carlopi wants to merge 7 commits into
duckdb:v2.0-cyanopterafrom
carlopi:connect_passthrough
Open

Connect passthrough#25990
carlopi wants to merge 7 commits into
duckdb:v2.0-cyanopterafrom
carlopi:connect_passthrough

Conversation

@carlopi

@carlopi carlopi commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Three changes:

  1. A new matcher that match a token (StatementToken)
  2. A new grammar, that looks like:
Statement            <- DisconnectStatement / PassthroughStatement
PassthroughStatement <- StatementToken+
  1. Each AttachedDatabase has a connected_grammar, that when set is used when CONNECT-ed to that specific database

These pieces allow, once connected, to pass down any grammar for that to be evaluated on the remote/server side.

Idea is that you can do:

CONNECT '<...>';
ARBITRARY STRING;  --- sent and evaluated by remote (possibly will fail or not)
DISCONNECT;             --- evaluated locally

with ARBITRARY STRING just be passed down.

See in particular test/sql/connect/connect_passthrough_quack.test that demo (via quack) that everything is passed down to the server.
This is particularly relevant for postgres where syntax might / will differ, and might not be recognized by DuckDB parser.

@carlopi
carlopi requested a review from Tishj September 22, 2026 07:18
Tishj
Tishj reviewed Sep 22, 2026
View reviewed changes
Comment thread src/include/duckdb/main/capi_v2/capi_v2_internal.hpp
@carlopi carlopi mentioned this pull request Sep 22, 2026
Merged
Tishj
Tishj reviewed Sep 22, 2026
View reviewed changes
Comment thread src/include/duckdb/parser/peg/matcher/statement_token_matcher.hpp Outdated
Tishj
Tishj reviewed Sep 22, 2026
View reviewed changes
Comment thread src/parser/peg/compiled_grammar.cpp Outdated
Tishj
Tishj reviewed Sep 22, 2026
View reviewed changes
Comment thread src/include/duckdb/main/client_config.hpp
//! The compiled grammar active for the connection
shared_ptr<CompiledGrammar> cached_grammar;
//! The grammar of the database this client is CONNECT-ed to; unset when not connected
shared_ptr<CompiledGrammar> connected_grammar;

@Tishj Tishj Sep 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just reuse cached_grammar for this? We'll just have to save the original current_dialect setting on CONNECT and reinstate the cached_grammar on DISCONNECT

@Tishj Tishj Sep 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah actually, no that's for grammar extensions.
DialectExtension (current_dialect) goes through client_config.current_dialect and does a look-up of the dialect-extension to return its grammar

@Tishj Tishj Sep 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the existing current_dialect setting for this instead? You can create a passthrough dialect, reserve that name by not allowing it through regular SET current_dialect methods and then everything else should just work.
That also eliminates the set/get connected-grammar callbacks, those can just override the "passthrough" name with another dialect extension they register.

You'll only need to preserve and re-instate the existing current_dialect setting on CONNECT and DISCONNECT respectively

evertlammerts pushed a commit to evertlammerts/duckdb that referenced this pull request Sep 22, 2026
@Mytherin
Pin the StatementType count in the C API (duckdb#26003)
d4e7256 
The v2 C API surfaces StatementType under the same numeric values, but
core had no count sentinel, so a member appended in core could only be
caught by two runtime tests probing the values past the last known one.

STATEMENT_TYPE_COUNT makes that a static_assert: appending a member
shifts the count and fails to compile until the new type has an id in
the v2 spec. The two probe tests are no longer needed.

Originally first commit at duckdb#25990,
but as pointed out better reviewed independently.

Newer version of duckdb#25972
@carlopi
Add a PassthroughStatement holding unparsed SQL
e0edeb5 
A PassthroughStatement carries the source text of a statement that was
never handed to the parser, so it can be forwarded verbatim to another
engine. The text lives in the base SQLStatement::query field.

The C API mirrors StatementType numerically, so the new member is pinned
in the v2 spec as well. Nothing produces a PassthroughStatement yet.
@carlopi
Add a matcher for a token the parser does not interpret
3f77728 
StatementTokenMatcher consumes any single token that is not a statement
boundary. Repeating it covers a statement whose syntax this parser does
not know - SQL meant for another engine - while the tokenizer keeps
deciding where the statement ends, so a ';' inside a string, a quoted
identifier or a dollar-quoted body does not split it.

The grammar cannot express this: '!' is silently dropped by the PEG
parser, '.' is not a rule operator, and the operator matcher does not
cover '(' or ','.

No grammar rule uses it yet.
@carlopi
Add the grammar used while CONNECT-ed
df5bdb8 
The passthrough grammar replaces the Statement rule with

  Statement            <- DisconnectStatement / PassthroughStatement
  PassthroughStatement <- StatementToken+

so a client that is CONNECT-ed interprets only DISCONNECT and hands
every other statement to the remote verbatim. It keeps the base
tokenizer, so statement boundaries stay DuckDB's, and the statement
text is filled in from the parse result's extent like any other
statement.

The grammar is compiled once per database, next to the base grammar.
Nothing selects it yet.
@carlopi
Let the database being connected to choose the grammar
b3cd296 
An AttachedDatabase carries the grammar used for statements issued while
a client is CONNECT-ed to it. It defaults to the passthrough grammar, so
a remote gets statements verbatim and decides itself what is valid; a
backend that speaks DuckDB SQL can override it and have those statements
parsed locally as before.

CONNECT installs that grammar, DISCONNECT clears it, and the resolver
keeps reading only ClientConfig.

Two consequences, both intended. CREATE GRANT against a quack server now
fails on the server rather than in the local parser. CONNECT is itself
forwarded, so connections chain over several hops - a client that cannot
even load the postgres extension can reach Postgres through a hop that
can. DISCONNECT stays local, so the client can always get back.
@carlopi
Test that CONNECT chains over two hops
051be24 
The client attaches to hop A and connects; the CONNECT that follows is
forwarded, so hop A is what interprets it and connects onwards to hop B.
A query then travels the whole chain, and DISCONNECT brings the client
straight back to local.

This is what forwarding CONNECT is for: a client that cannot speak - or
even load - the backend it ends up talking to can still reach it. The
test needs real hops in separate processes, so it lives with the shell
tests rather than in sqllogictest.
@carlopi
Fix test/sql/connect/connect_passthrough_quack.test via explicit load
2fa48ff
@carlopi
Review feedback: mutex is enough, and rename to PassthroughTokenMatcher
af0d4a9
@carlopi
carlopi force-pushed the connect_passthrough branch from f3062d1 to af0d4a9 Compare September 22, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@Tishj Tishj Tishj left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Member

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@carlopi @Tishj