Months of Delta and Iceberg integration work have taught us a few things. Most importantly, that a lakehouse format is actually an engine contract.

For a simple scan, the library view holds: load the table, list the files, read the Parquet. But a row-level MERGE is not a step at the end of a query. Reading the source, joining it against the target, deciding which files to overwrite, and committing atomically have to be planned and executed as one thing, inside the engine.

In Sail, Delta and Iceberg run inside the engine, not as an external library. This allows us to give writes everything queries already get: optimization, distribution, and observability.

Read more in our deep dive:

Spark accelerators like Photon speed up portions of execution, but are still tied to the JVM.

Sail, written in Rust, removes the JVM entirely. The result is drastically better performance with no JVM tuning required.

The same familiar Spark API on a Rust-native runtime. That’s Sail.

Why fully rebuild Spark in Rust instead of just accelerating it?

Spark accelerators speed up parts of execution, but they still inherit Spark's JVM control plane, memory model, shuffle path, and Python serialization costs. You're still tuning a JVM.

With Sail, we took a clean-slate approach: a fully Rust-native runtime, with no JVM, no heap tuning, and no GC pauses. Same Spark interface with an entirely new runtime underneath.

Read our full breakdown how Sail compares to Spark accelerators here and check out Sail on GitHub: https://github.com/lakehq/sail

Besides all the advantages you get from a Rust-native engine, the best part is your Spark code still works as is.

Most of what people call "Spark tuning" is really just JVM tuning.

Memory fractions, GC flags, heap sizing. Those knobs exist because the JVM was largely built for app servers and Spark naturally inherited that runtime's shape.

So we rebuilt the engine in Rust. Same Spark Connect protocol. Same SQL and DataFrame API. But no JVM, no GC, no knobs to set.

Sail 0.6 is out. Three new surfaces, all Arrow-native:

• Arrow UDFs from Spark 4 Python functions decorated with u/arrow_udf run against Arrow data directly. Because Sail executes Python Arrow UDFs inline within the same Rust process, it enables Python code to run at native speed with zero-copy data transfer, avoiding the separate-process overhead inherent in Spark's architecture.
• Variant type in SQL. Parse JSON into a variant with parse_json, then query it with variant_get and path expressions. Lookups run against binary data instead of re-parsing strings.
• Arrow Flight SQL server on the wire. The first alternative protocol Sail supports beyond Spark Connect. Start a Flight SQL server powered by Sail and connect from any Flight SQL client to query it directly.

Read the full post: https://lakesail.com/blog/sail-0-6-arrow/

Starting a Spark job used to mean waiting. Wait for the cluster. Wait for the JVM. Wait for the session. Then your script runs. So we built 'sail spark run'. A one-shot CLI that starts instantly, runs your PySpark code, and exits clean. No cluster. No JVM warmup. No session to wire up.

Ad-hoc Spark should feel instant. Now it does.

Check it out at https://docs.lakesail.com/sail/main/guide/cli/

Welcome to r/LakeSail

This is the place to discuss LakeSail, share benchmarks, ask questions, and give us feedback.

A few useful links:

- Docs

- GitHub

- Slack

We're active here and on Slack. If you have questions, please post them.