LakeSail

Spark accelerators like Photon speed up portions of execution, but are still tied to the JVM. Sail, written in Rust, removes the JVM entirely. The result is drastically better performance with no JVM tuning required. The same familiar Spark API on a Rust-native runtime. That’s Sail.

Why fully rebuild Spark in Rust instead of just accelerating it? Spark accelerators speed up parts of execution, but they still inherit Spark's JVM control plane, memory model, shuffle path, and Python serialization costs. You're still tuning a JVM. With Sail, we took a clean-slate approach: a fully Rust-native runtime, with no JVM, no heap tuning, and no GC pauses. Same Spark interface with an entirely new runtime underneath. Read our full breakdown how Sail compares to Spark accelerators here → https://lnkd.in/gwyF8Fq2

I'm excited to announce the panel for our next Data for AI on Jun 3rd at Yes SF! We have four experts in the field, who will be speaking on our panel on unifying enterprise-wide data for agents, each covering various dimensions of this challenge at scale. Josh W., LiveRamp: Josh is a Principal Architect at LiveRamp, where he's at the intersection of AI, data and MarTech. In case you missed his last Data for AI presentation, he's built a semantic middle layer that allows LiveRamp's agents to understand the context of data from over a dozen systems. They had to balance cost, security, auditability, and many other factors to make this initiative a success. Previously, he's held senior roles at Highnote, Coinbase, Twilio, and Salesforce, and holds a patent on database server access management. Mark Hoerth, Datastrato: Mark leads product at Datastrato, working on Apache Gravitino and the next chapter of open table formats for AI. He joined from Dremio, where he held product and solution architecture roles spanning Apache Iceberg lakehouse deployments and AI semantic search, and led Dremio's efforts security, Iceberg, and Apache Polaris. A Stanford alum based in the Bay Area, Mark is a longtime Silicon Valley startup builder. Andrew Madson, Fivetran: Andrew leads Developer Relations at Fivetran, where he builds programs that help developers and data teams adopt modern data and AI tooling. He's the author of O'Reilly's Apache Polaris: The Definitive Guide, with two more books on the way — AI-Ready Data (Wiley) and Data Transformation (O'Reilly). Andrew previously built DevRel functions at Tobiko Data and Dremio, and he teaches data science and engineering as a graduate professor. Alexy Khrabrov, LakeSail: Dr. Khrabrov is the Head of Community at LakeSail, building the Spark-compatible AI Lakehouse of tomorrow in Rust. He is also the founder of the Community Research Center for Reliable AI at Northeastern University, founder and organizer of AI By the Bay, Bay Area AI, AI Agent SF, the longest-running, deepest technical OSS AI communities, conferences, and meetups in the San Francisco Bay Area. Previously, Alexy was the Director of Open-Source Science at IBM Research, AI Community Architect at Neo4j, Senior Software Engineer at Amazon, and a co-founder and engineer in several Bay Area startups. Spaces are still available, but are running out! Sign up today! https://luma.com/8tvd2xla Thank you to our sponsors, Datastrato, Fivetran & LakeSail for making this event possible! See you there!

The Apache Iceberg meetup was a blast! Shehab Amin and I gave an overview of LakeSail and its Sail support of iceberg via Engine Contract. A really strong engineering audience, that stayed focused through 2.5 hours of deep technical talks. Thank you Scott Haines for the masterful photo. We’ll be back!

Come to the Bay Area Apache Iceberg meetup tonight! Shehab Amin and I will talk about LakeSail, a Spark ecosystem rebuilt in Rust, and how our Iceberg support upholds an Engine Contract, building support for Iceberg into the engine. Thank you for hosting us, Zhenni Wu, Scott Haines, Lisa N. Cao, Nathan Yee! https://lnkd.in/gyTnEHyD

What if Spark was rewritten in Rust? Well, that's Sail. Besides all the advantages you get from a Rust-native engine, the best part is your Spark code still works as is.

Apache Spark is an API for big data, and Sail from LakeSail is a new implementation for the Agentic AI era.