DuckDB CLI does not load persistent secrets on startup #18021
Description
What happens?
The documentation for secrets says:
On startup of DuckDB, persistent secrets are read from this directory and automatically loaded.
However, it seems that persistent secrets are only loaded after a related command is run, such as creating a new secret or running FROM duckdb_secrets();.
To Reproduce
First, create a persistent secret in one CLI instance:
$ duckdb
DuckDB v1.3.1 (Ossivalis) 2063dda3e6
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
CREATE OR REPLACE PERSISTENT SECRET minio (TYPE s3, PROVIDER config, KEY_ID 'redacted', SECRET 'redacted', ENDPOINT 'localhost:9000', USE_SSL false, URL_STYLE path);┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
Then, in a new CLI instance, attempt to use the secret. First the query fails, but then succeeds after running VIEW duckdb_secrets():
$ duckdb
DuckDB v1.3.1 (Ossivalis) 2063dda3e6
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
SELECT count(*) FROM iceberg_scan('s3://path/to/data', version='00001-afc3775a-fa80-4d2d-aa07-669e6671b989');
Invalid Configuration Error:
Iceberg metadata file not found for table version '00001-afc3775a-fa80-4d2d-aa07-669e6671b989' using 'none' compression and format(s): 'v%s%s.metadata.json,%s%s.metadata.json'
FROM duckdb_secrets();┌─────────┬─────────┬──────────┬────────────┬────────────┬──────────────────────┬─────────────────────────────────────────────────────────┐
│ name │ type │ provider │ persistent │ storage │ scope │ secret_string │
│ varchar │ varchar │ varchar │ boolean │ varchar │ varchar[] │ varchar │
├─────────┼─────────┼──────────┼────────────┼────────────┼──────────────────────┼─────────────────────────────────────────────────────────┤
│ minio │ s3 │ config │ true │ local_file │ ['s3://', 's3n://'… │ name=minio;type=s3;provider=config;serializable=true;… │
└─────────┴─────────┴──────────┴────────────┴────────────┴──────────────────────┴─────────────────────────────────────────────────────────┘
SELECT count(*) FROM iceberg_scan('s3://path/to/table', version='00001-afc3775a-fa80-4d2d-aa07-669e6671b989');┌──────────────┐
│ count_star() │
│ int64 │
├──────────────┤
│ 744473 │
└──────────────┘
OS:
MacOS, aarch64
DuckDB Version:
v1.3.1 (Ossivalis) 2063dda
DuckDB Client:
CLI
Hardware:
No response
Full Name:
Patrick Lucas
Affiliation:
Atomic Wire
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?
- Yes, I have