Why it matters
Development and test workloads can usually tolerate a slower run when a Spot worker is reclaimed, and workers are most of a cluster’s EC2 cost.How nOps finds it
nOps reads the last 7 days of on-demand EC2 billing for instances launched by Databricks (SQL warehouses excluded), grouped by job, instance pool or cluster. A group is flagged when at least 90% of its cost is non-production, judged from the resource’s own environment or lifecycle tag, then the environment nOps assigns. One instance per cluster per hour is kept on demand as the driver. The saving is the remaining worker cost times your Spot discount for that instance type and region: your own billed Spot rate when you ran at least 24 Spot hours of it, otherwise the median current Spot price across the region’s Availability Zones from AWS’s public Spot price history. Groups already mostly on Spot or mostly covered by commitments, and savings under $25 a month, are left out.Before you start
Permission to edit the cluster, job or pool configuration in Databricks.How to implement
- Confirm the workload is non-production and can tolerate losing a worker during a run.
- Edit the compute and, under Advanced → Instances, turn on Spot instances for workers. In the API or Terraform, set
aws_attributes.availabilitytoSPOT_WITH_FALLBACKandfirst_on_demandto 1 so the driver stays on demand. - For an instance pool, set the pool to Spot and give clusters an on-demand driver pool.
- Run the job and check run time and failures.
- Put the setting in your cluster policy or Databricks Asset Bundle so new non-production clusters use it.
Risks and considerations
- Spot workers can be reclaimed. Databricks replaces them and falls back to on-demand, but runs can take longer.
- Keep the driver on demand; losing the driver fails the run.
- To roll back, set availability back to
ON_DEMAND.
FAQs
Why does nOps keep one instance on demand?
Why does nOps keep one instance on demand?
Each cluster has a driver node. If a Spot driver is reclaimed the whole run fails, so the estimate keeps one instance per cluster on demand.