LakeSail

We've spent two amazing days with Yann LeCun and Roberto Di Cosmo in Paris, at the FPT Software office, partners of Christopher Cuong T. Nguyen, the leading pillar of the indusrtial AITOMATIC group and core member of The AI Alliance. Yann and Roberto signify the best Europe contributes to AI: -- rigorous STEM background -- deep appreciation of history, cultural, scientific, and software -- multicultural sensibility by design We are putting together an OSS AI Semantic Layer at querygraph.ai that will power Agentic AI data quality and discovery layer, supporting the Al Alliance Tapestry project we kicked off and Open Data Spaces by Mitch Tsuda and the Government of Japan. Slava Tykhonov, Simon Hodson, and CODATA consortium are leading the way for Responsible AI acccountable to the people and governments of the world. Let's join forces for the Responsible AI of tomorrow! And we at LakeSail are working hard to implement an AI-ready Data Lake with the catalogs that will support Responsible AI Data Discovery and Interoperability. If you are an LLM company, a hyperscaler, or an Agentic AI startup, you can rely on our Rust-based, infinite data layer to be your system of record, your ground truth, and your OSS AI foundation for auditable, performant, and trusted AI.

Today, for the first time in 15 years, I launched a Spark shell without having Spark JVM on my computer. The future is Rust. LakeSail

Most of what people call "Spark tuning" is really just JVM tuning. Memory fractions, GC flags, heap sizing. Those knobs exist because the JVM was largely built for app servers and Spark naturally inherited that runtime's shape. So we rebuilt the engine in Rust. Same Spark Connect protocol. Same SQL and DataFrame API. But no JVM, no GC, no knobs to set.

Sneak peek at what’s coming with the #OneLake Catalog 👀 This is what the power of open standards looks like data converted to Iceberg metadata from Parquet using #pyiceberg tests using #lakesail reading the results using #duckdb I think the best thing about iceberg is the Rest Catalog 😁 in a sense it feels like a database The more things change, The more they stay the same #Lakehouse #C++ #Rust #MicrosoftFabric #ApacheIceberg #SQL

Most of Spark's "performance ceiling" today is just the JVM it's built on. We decided to rewrite Spark in Rust, replacing the JVM entirely. The technical advantages of this decision speak for themselves.

I’ve been waiting for this moment for three years 🚀 This is, technically, the first #ApacheIceberg table written using the #OneLake catalog REST API outside of engineering and partners 🤯 It’s coming soon 👀 Huge thanks to Matthew Hicks for the early access 🙏 and Kevin Liu for the work on PyIceberg 💪 What a day 🔥 it works out of the box with #lakesail too, which was a very nice surprise #onelake #MicrosoftFabric #Lakehouse #rust #apachedatafusion #Python

Every data engineer has had this moment: It's midnight. A Spark job is spilling 100+ GB to disk. GC pauses are stacking. The cloud bill is climbing. You stare at the JVM heap and wonder if there's a better way. There is. It's written in Rust. LakeSail is an open-source, drop-in replacement for Apache Spark. Not a fork. Not a wrapper. A ground-up Rust rewrite that speaks the Spark Connect protocol — so your existing PySpark code runs without a single line changed. What that bought them: ↳ ~4x faster execution (8x on some workloads) ↳ 94% lower cloud cost ↳ Zero shuffle spill vs. 110+ GB on vanilla Spark ↳ No JVM. No GC. No serialization overhead on Python UDFs. The architecture: Apache Arrow for columnar in-memory data, DataFusion for query planning, Rust's ownership model for memory safety that the JVM could never promise. Spark served its generation. Sail feels like what the next one is going to run on. https://lnkd.in/g3eGFiYx ⭐ GitHub: github.com/lakehq/sail If you could cut your data infra bill by 90% without touching a single line of code — what would you do with the savings? Obviously not paying the Engineers... #BigData #Rust #ApacheSpark #DataEngineering #OpenSource

Sail 0.6 is out. Three new surfaces, all Arrow-native: - Arrow UDFs from Spark 4. Python functions decorated with @arrow_udf run against Arrow data directly. Because Sail executes Python Arrow UDFs inline within the same Rust process, it enables Python code to run at native speed with zero-copy data transfer, avoiding the separate-process overhead inherent in Spark's architecture. - Variant type in SQL. Parse JSON into a variant with parse_json, then query it with variant_get and path expressions. Lookups run against binary data instead of re-parsing strings. - Arrow Flight SQL server on the wire. The first alternative protocol Sail supports beyond Spark Connect. Start a Flight SQL server powered by Sail and connect from any Flight SQL client to query it directly. Read the full post: https://lnkd.in/gdAJ28gw

Doing my part to get the jvm out of the modern data stack 🦀! Excited to share my first rust oss contribution to LakeSail, a drop-in spark replacement built on datafusion and arrow. Rust is tough, so big shoutout to Heran Lin and David López González for their guidance. More to come! https://lnkd.in/exjSfS3i

Starting a Spark job used to mean waiting. Wait for the cluster. Wait for the JVM. Wait for the session. Then your script runs. So we built 'sail spark run'. A one-shot CLI that starts instantly, runs your PySpark code, and exits clean. No cluster. No JVM warmup. No session to wire up. Ad-hoc Spark should feel instant. Now it does.