The Lakehouse Format Is an Engine Contract

Delta Lake and Apache Iceberg have become the backbone of the modern lakehouse. They give plain object storage the advanced capabilities of a data warehouse: ACID transactions, time travel, schema evolution, and row-level updates. These formats are ubiquitous in production data pipelines nowadays.

As an open-source, Rust-native compute engine with Apache Spark compatibility, Sail brings scalable data processing to where most of our users’ data already lives. The question was never whether Sail should support these formats, but how deeply.

Many engines can read a snapshot of a Delta Lake or Iceberg table, but struggle when it comes to updates to an existing table, let alone doing so in a performant fashion. The limitations are often due to the level of integration. To support a lakehouse format fully, we have to treat it as a contract the engine honors across the whole stack, from scan planning to the atomic commit.

This is the approach we take for Sail. We built Delta Lake and Iceberg into Sail’s own planner and runtime, so an operation such as MERGE is represented as an organic execution plan, visible to the optimizer and scheduled across workers like everything else.

How Deep the Contract Reaches

It is tempting to treat a lakehouse format as a storage library: load a table, list some files, and read the Parquet data. For a simple scan, that view holds. When it comes to partition pruning, statistics, deletion vectors, and column mapping, the situation becomes complex. For writing, especially when modifying an existing table, things get even trickier. To operate Delta Lake or Apache Iceberg correctly, a lot of details in the format specification have to permeate through the whole engine:

• Snapshots, manifests, and delete files
• Scan planning and file pruning
• Write planning, manifest generation, and optimistic commit
• Schema and partition evolution
• Catalog and metadata-pointer ownership

In our view, a lakehouse format is an engine contract. The integration happens at all levels, from the client protocol to object storage. The reason is twofold. Firstly, many DML operations such as MERGE do not happen at the end of query execution. All the steps, including reading the source table, joining the source and target tables, deciding which files to overwrite on the target table, and finally committing the target table, have to be planned and executed together, especially when distributed processing is involved. Secondly, query planning itself often involves computation so the format requires a performant compute engine for efficient processing of manifests, which becomes critical when the table manages millions of files or a long history yet to be compacted. Therefore, the semantics of the full specification cannot be abstracted away by a connector library that owns read and write. The engine has to make the operation executable.

This is not something we understood from day one. When we first started the work around Delta Lake a year ago, we depended on the delta-rs and delta-kernel-rs libraries but ultimately removed these dependencies entirely as we built logical and physical planning capabilities deeply within our codebase. The lessons were then adopted for our Iceberg integration so that iceberg-rust was not a dependency of Sail. We are still thankful for these upstream projects, as we adopted some code around the format specification under their Apache license.

Example: Conditional Deletes

It may sound crazy that we decided to own everything around Delta Lake and Iceberg within Sail’s codebase, but the outcome is a natural fit into our query planning and execution architecture. Let’s consider how it works for the conditional delete operation for Delta Lake. A DELETE FROM ... WHERE ... statement partially overwrites an existing table. There are usually two strategies for this operation. One is copy-on-write, which scans existing data files and produces new files with the deleted rows filtered out. A new table commit tracks which files are added and removed. On a Delta table with deletion vectors enabled, the other strategy, merge-on-read, is involved. It finds the files that hold the matched rows, writes a bitmap marking the deleted positions, and keeps the data file untouched. The table commit tracks the added deletion vector file.

A merge-on-read delete, planned as the engine’s own nodes: find the touched files, write deletion vector bitmaps, reuse the data files, and then commit.

An EXPLAIN statement would show all these nodes in the physical plan instead of having everything as one black box. Note that we haven’t talked about distributed processing, but it just happens naturally. This is because all the nodes are just ordinary physical operations, so they are partitioned and scheduled across workers by design. This means finding the affected file and writing the deletion vectors can be parallelized, with an atomic commit that runs as a single-partition operation at the end. Scalability is not an issue here as the computation is not confined by a single-host compute engine.

Where We Are and What’s Next

Right now, we support a rich set of features for Delta Lake and Iceberg, including common DDL and DML statements. The reader supports pruning based on partitions and statistics, and understands lakehouse concepts such as deletion vectors, column mapping, and time travel. The writer allows you to write new tables, insert data into existing tables, or overwrite an entire table using SQL or Spark DataFrame APIs.

Besides accessing tables by the storage path, you can use Delta Lake and Iceberg for all these catalogs that Sail supports: Iceberg REST, Unity, AWS Glue, Hive Metastore, OneLake, and an in-memory catalog for local work.

MERGE for Iceberg and UPDATE for both formats are the remaining DML statements to be supported. All these will be delivered as we refine our foundation of row-level operations.

We’re also planning advanced catalog features. The Iceberg REST protocol defines server-side scan planning, where the catalog plans the scan and returns the file tasks to the engine. Unity Catalog also enhanced its support for catalog-managed tables recently. Supporting these features would make Sail a better choice for your lakehouse needs.

Try It Yourself

Getting started with the lakehouse is quick and easy in Sail. Install the pysail Python package, and follow the documentation to access Delta Lake or Iceberg tables using the familiar PySpark APIs over the Spark Connect protocol.

This is the first post about Sail’s lakehouse work, and we expect to write more around these topics! If you’d like to follow our journey, join our Slack community to ask questions and drop us a GitHub star!

If you want managed Sail with governance, observability, and enterprise controls, sign up for the LakeSail Platform.

Delta Lake™ is a trademark of LF Projects, LLC. Apache® and Apache Iceberg™ are trademarks (or registered trademarks) of The Apache Software Foundation. LakeSail, Inc. is an independent company and is not affiliated with, endorsed by, or sponsored by the Delta Lake project, LF Projects, LLC, or The Apache Software Foundation.