Big Data Processing for the AI Era

Sail: An open-source computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.

Sail 0.2.2: 80% of Spark functionality with an in-house SQL parser in Rust. Read more

The LakeSail Slack Community is now live. Join now

Apache Spark → Sail

Processing Speed

94% ↓

Hardware Costs

Code Changes

The Sail architecture is designed to shape the future of distributed processing.

Sail features a drop-in replacement for Spark SQL and the Spark DataFrame API in both single-host and distributed settings.

bash

pip install "pysail[spark]"

bash

sail spark server --port 50051

python

from pysail.spark import SparkConnectServer

server = SparkConnectServer(port=50051)
server.start(background=False)

bash

kubectl apply -f sail.yaml
kubectl -n sail port-forward service/sail-spark-server 50051:50051

Once you have a running Sail server, you can connect to it in PySpark.
No changes are needed in your PySpark code!

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()
spark.sql("SELECT 1 + 1").show()

LakeSail offers commercial support for Sail, with flexible coverage tailored to your needs. Get in touch for more details.

Free

Negotiable