Skip to main content
Databricks on AWS can launch worker nodes as EC2 Spot instances, which cost much less than on-demand instances, and fall back to on-demand when Spot is not available. nOps finds non-production Databricks jobs, pools and clusters that run their workers on demand and estimates the EC2 saving from running the workers on Spot while the driver stays on demand.

Why it matters

Development and test workloads can usually tolerate a slower run when a Spot worker is reclaimed, and workers are most of a cluster’s EC2 cost.

How nOps finds it

nOps reads the last 7 days of on-demand EC2 billing for instances launched by Databricks (SQL warehouses excluded), grouped by job, instance pool or cluster. A group is flagged when at least 90% of its cost is non-production, judged from the resource’s own environment or lifecycle tag, then the environment nOps assigns. One instance per cluster per hour is kept on demand as the driver. The saving is the remaining worker cost times your Spot discount for that instance type and region: your own billed Spot rate when you ran at least 24 Spot hours of it, otherwise the median current Spot price across the region’s Availability Zones from AWS’s public Spot price history. Groups already mostly on Spot or mostly covered by commitments, and savings under $25 a month, are left out.

Before you start

Permission to edit the cluster, job or pool configuration in Databricks.

How to implement

  1. Confirm the workload is non-production and can tolerate losing a worker during a run.
  2. Edit the compute and, under Advanced → Instances, turn on Spot instances for workers. In the API or Terraform, set aws_attributes.availability to SPOT_WITH_FALLBACK and first_on_demand to 1 so the driver stays on demand.
  3. For an instance pool, set the pool to Spot and give clusters an on-demand driver pool.
  4. Run the job and check run time and failures.
  5. Put the setting in your cluster policy or Databricks Asset Bundle so new non-production clusters use it.

Risks and considerations

  • Spot workers can be reclaimed. Databricks replaces them and falls back to on-demand, but runs can take longer.
  • Keep the driver on demand; losing the driver fails the run.
  • To roll back, set availability back to ON_DEMAND.

FAQs

Each cluster has a driver node. If a Spot driver is reclaimed the whole run fails, so the estimate keeps one instance per cluster on demand.