Replies: 7 comments 6 replies
-
|
Hmm, interesting! What are the use cases you want to enable with that? At some point, DuckDB will need to execute and materialize the results as Arrow format, since we don't use Arrow internally. Or would an Arrow recordset/scanner work? We can read those, but I don't think we write them... |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply, I should have included more context, and maybe my question doesn't make sense in the context of polars. My basic use case is for students to write more pythonic 'piped' syntax we see in |
Beta Was this translation helpful? Give feedback.
-
|
Hi, It would be useful to use LazyDataFrame without loading all data into memory. For instance : duckdb.sql("SELECT * FROM large_data").lazy().filter(pl.col("A") == 4).select(pl.col("A") + 10).limit(10).collect() |
Beta Was this translation helpful? Give feedback.
-
|
Looking for the same thing! This would open the door to a really strong duckdb+delta integration, because polars has a |
Beta Was this translation helpful? Give feedback.
-
|
Following up here, with the announcement of Ducklake, there's an even stronger case for this integration. Needing Polars to add a read_ducklake before adopting would be a bit unfortunate, but if duckdb supported a |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to create a lazyframe from a record batch reader? |
Beta Was this translation helpful? Give feedback.
-
|
The attention here is misplaced, this is not a DuckDB limitation, we can create a record batch reader that lazily scans a duckdb relation just fine, but Polars does not expose any way to lazily construct a LazyFrame, all of the ingestion methods eagerly materialize the input Please raise awareness on the Polars side if you want this feature, we'd be happy to implement this when they support it. |
Beta Was this translation helpful? Give feedback.
-
|
I think the IO plugins within polars enable what is asked for - explicitly supporting various pushdowns and enabling incremental materialization, https://docs.pola.rs/user-guide/plugins/io_plugins/ here is a quick implementation someone did: duckdb/ducklake#121 (comment) although the specific example someone created for ducklake uses the .pl() still, going via record batches and more incrementally returning is also possible as shown in the polars documentation for io plugins specifically |
Beta Was this translation helpful? Give feedback.
-
|
I'd like to see an example of "going via record batches and more incrementally returning", I couldn't find it in the link you shared |
Beta Was this translation helpful? Give feedback.
-
|
In the example CSV reader in the polars documentation they are returned incrementally in batches, here is a slight modification to the duckdb reader which uses record batches: def duckdb_source(connection: duckdb.DuckDBPyConnection, table: str, extra: str = "") -> pl.LazyFrame:
"""
A polars IO plugin for DuckDB.
"""
query = f"select * from {table} {extra} limit 0"
schema = connection.sql(
query
).pl().schema
def source_generator(
with_columns: list[str] | None,
predicate: pl.Expr | None,
n_rows: int | None,
batch_size: int | None,
) -> Iterator[pl.DataFrame]:
if batch_size is None:
batch_size = 512
if with_columns is None:
cols = "*"
else :
cols = ",".join(with_columns)
query = f"select {cols} from {table} {extra}"
if predicate is not None:
query += f" where {_predicate_to_sql(predicate)}"
if n_rows is not None:
query += f" limit {n_rows}"
logger.debug(query)
results = connection.sql(query).fetch_arrow_reader(batch_size)
while True:
try:
yield pl.from_arrow(results.read_next_batch())
except StopIteration:
break
return register_io_source(source_generator, schema=schema) |
Beta Was this translation helpful? Give feedback.
-
|
I did some more digging, should have responded, it makes sense to me 👍 |
Beta Was this translation helpful? Give feedback.
-
|
This has been implemented in #17947 that will land in v1.4.0 |
Beta Was this translation helpful? Give feedback.
-
|
we are so back ... |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
duckdb's integration with polars is really nice.
I'd really like to be able to use
polarspythonic notation withduckdb's superior performance with large, out-of-RAM data (including remote data, like partitioned parquet files on S3 buckets). Based on the current documentation though, it looks like the.pl()method creates an in-memory dataframe, not a lazy data frame. Is it possible to generate a lazy data frame instead?Beta Was this translation helpful? Give feedback.
All reactions