Agent Skills for Spark Workloads

Sail is built for the agentic era. Our mission is to bring unified compute to both humans and agents. We envision a future where millions of lightweight jobs run in parallel as your agents explore complex realities and act at scale. Compute scales up and down in real time with demand, giving you instant efficiency without wasted spend. Idle resources never inflate your cloud bill.

Last year, we introduced the Sail MCP server, which exposes Sail tools through the Model Context Protocol. Now, as agents become more capable and more useful, we are introducing a “one-shot” execution mode in the Sail CLI. This allows your agent to operate as a skilled data and AI engineer, powered by the lightning-fast Sail engine.

Starting with Sail 0.5.3, you can use the sail spark run command to run any PySpark script without explicitly provisioning a server. Here is an example:

echo 'spark.sql("SELECT 1 + 1").show()' | sail spark run

The script can access the Spark session through the spark variable, which connects to a local Sail server using the Spark Connect protocol. The Sail server starts instantly when you run the sail spark run command and automatically stops when the script finishes.

The command is useful for running a single SQL query, quickly trying out Sail features, or validating a small PySpark snippet. Its real beauty lies in its composability, grounded in Unix philosophy. The command accepts a script from either stdin or a file, and within the script you have full control over where the output goes, whether to stdout, files, or cloud storage.

Your agent is already a Spark expert, trained on a vast corpus of examples for the PySpark DataFrame API and Spark SQL. But a typical Spark cluster can take minutes to spin up, making it a poor fit for an agent’s ad hoc needs. That is where sail spark run comes in. By writing a SKILL.md file with instructions for the Sail CLI, you can equip your agent with data and AI engineering capabilities powered by Sail. Each script bootstraps within a second and finishes much faster. Sail makes this possible by acting as a high-performance, resource-efficient drop-in replacement for Spark behind the scenes. The result is a natural fit for agentic workflows with frequent data and AI skill invocations. You can see a SKILL.md example in the Sail documentation.

We hope you and your agents enjoy this new feature. Join our Slack community, and follow our latest developments on GitHub!