How Sail Compares to Photon and Other Spark Accelerators

A question we tend to get a lot when people first hear about Sail is: How does Sail compare to Spark accelerators like Databricks’ Photon, DataFusion Comet, and Velox? And why create an entirely new engine instead of contributing to the existing codebase?

To answer this, it’s important to first clarify what Spark accelerators like Photon, Comet, and Velox actually do.

How Spark Accelerators like Photon, Comet, and Velox Work

Spark accelerators such as Photon, Comet, and Velox (via Gluten) integrate with JVM-based Spark to replace portions of the physical execution layer with optimized native implementations. They often execute operators in native code and commonly use off-heap columnar data to reduce JVM GC pressure, while relying on Spark’s JVM-based control plane for planning and orchestration. This hybrid architecture creates a strict boundary between native operator execution and Spark’s JVM-level runtime. Consequently, executing custom code typically requires crossing back into JVM or Python execution contexts (often involving data serialization, format conversions, or other handoffs), which helps explain why Photon does not support Python UDFs, while Sail supports them natively.

In fact, Databricks’ documentation on Photon lists several additional limitations:

• Photon doesn’t support UDFs, RDD APIs, or Dataset APIs.
• Stateful streaming is not supported. Photon supports stateless streaming only.
• Photon doesn’t improve queries that normally run in under two seconds.

Databricks, “What is Photon?” Databricks Documentation, accessed May 28, 2026.

Why Sail

Spark was designed 17 years ago. Its foundations, a row-based internal data representation, JVM-based execution, and an architecture that predates the decoupling of storage and compute, made sense for the Hadoop era. They are now the constraints that limit what the system can do, especially for AI workloads.

To build the engine we set out to build, the JVM had to be eliminated. Not optimized around, not supplemented with native accelerators. Eliminated.

The JVM imposes constraints that propagate through every layer of a query engine: garbage collection pauses, object allocation overhead, Python interoperability through Py4J serialization, and heavy container footprints that limit cloud elasticity. Spark accelerators can speed up individual operators, but they cannot remove these constraints. They still run inside Spark’s JVM executors. The JVM remains the bottleneck.

Sail removes it entirely. Planning and execution run in a Rust-native query engine built on Apache Arrow’s columnar memory model and Apache DataFusion’s execution framework. Workers exchange Arrow batches directly using Arrow Flight rather than JVM-based shuffle. Python UDFs run in-process via PyO3 with zero-copy Arrow data sharing. Memory management relies on Rust’s compile-time ownership guarantees rather than JVM garbage collection. The process starts in seconds and consumes only tens of megabytes when idle.

This is also why Sail can serve as a unified engine rather than a specialized tool. JVM-based engines handle Python and AI/ML integration poorly, which forces organizations to maintain separate systems for data processing and AI workloads. Sail’s native Python support and Arrow-based memory model make it possible to run both in the same engine, without the serialization overhead that keeps these workloads apart.

Why Spark Compatibility

Upstreaming any of this to Spark would mean replacing its shuffle protocol, executor lifecycle, memory management assumptions, and Python bridge. These are components that every Spark connector, extension, and workload depends on. At that point, that is not a contribution to Spark, it is writing a whole different engine.

But Spark’s API is the lingua franca of distributed data processing. Millions of engineers know PySpark’s DataFrame and SQL interfaces. Rewriting an engine shouldn’t require rewriting every application that runs on it.

As a result, we decided that Sail should implement the Spark Connect protocol in Rust, giving users the same DataFrame and SQL APIs they already rely on. The interfaces stay unchanged. Underneath, execution runs on an entirely different architecture built for columnar processing, native Python interoperability, and elastic infrastructure.

No Spark. No JVM. But your Spark workloads run as is, all on a Rust-native runtime.

Benchmark Results

These architectural differences have proven to be significantly more performant. In the ClickBench benchmark, Sail outperforms every Spark + Spark accelerator configuration (Auron, Comet, Gluten-on-Velox) across all instance sizes (May 29, 2026).

Additionally, the results show that Sail outperforms Databricks’ and Snowflake’s own proprietary engines on significantly less hardware, not just Spark and Spark accelerators.

Getting Started

Getting started with Sail is simple. Head over to our deployment guide for a step-by-step guide on installing, configuring, and running Sail with PySpark.

Want to run Sail with managed infrastructure? LakeSail is launching a fully managed platform with built-in governance, observability, BYOC deployment, and enterprise controls. Get early access to see how Sail can improve performance and reduce costs for your team.