> ## Documentation Index
> Fetch the complete documentation index at: https://help.nops.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Spot Workers for Non-Production Databricks Compute

> Run the worker nodes of non-production Databricks clusters on Spot instances, keeping the driver on demand.

Databricks on AWS can launch worker nodes as EC2 Spot instances, which cost much less than on-demand instances, and fall back to on-demand when Spot is not available. nOps finds non-production Databricks jobs, pools and clusters that run their workers on demand and estimates the EC2 saving from running the workers on Spot while the driver stays on demand.

| | |
| - | - |
| **Cloud** | Databricks |
| **Category** | Compute |
| **Service** | Databricks compute on AWS |
| **Action** | Modify |
| **Effort** | Low |
| **Restart needed** | Yes |
| **Can be rolled back** | Yes |
| **Analysis window** | Last 7 days |
| **Source** | nOps |

## Why it matters

Development and test workloads can usually tolerate a slower run when a Spot worker is reclaimed, and workers are most of a cluster's EC2 cost.

## How nOps finds it

nOps reads the last 7 days of on-demand EC2 billing for instances launched by Databricks (SQL warehouses excluded), grouped by job, instance pool or cluster. A group is flagged when at least 90% of its cost is non-production, judged from the resource's own environment or lifecycle tag, then the environment nOps assigns. One instance per cluster per hour is kept on demand as the driver. The saving is the remaining worker cost times your Spot discount for that instance type and region: your own billed Spot rate when you ran at least 24 Spot hours of it, otherwise the median current Spot price across the region's Availability Zones from AWS's public Spot price history. Groups already mostly on Spot or mostly covered by commitments, and savings under \$25 a month, are left out.

## Before you start

Permission to edit the cluster, job or pool configuration in Databricks.

## How to implement

1. Confirm the workload is non-production and can tolerate losing a worker during a run.
2. Edit the compute and, under **Advanced → Instances**, turn on Spot instances for workers. In the API or Terraform, set `aws_attributes.availability` to `SPOT_WITH_FALLBACK` and `first_on_demand` to 1 so the driver stays on demand.
3. For an instance pool, set the pool to Spot and give clusters an on-demand driver pool.
4. Run the job and check run time and failures.
5. Put the setting in your cluster policy or Databricks Asset Bundle so new non-production clusters use it.

## Risks and considerations

* Spot workers can be reclaimed. Databricks replaces them and falls back to on-demand, but runs can take longer.
* Keep the driver on demand; losing the driver fails the run.
* To roll back, set availability back to `ON_DEMAND`.

## FAQs

> [accordion] Why does nOps keep one instance on demand?
>
> Each cluster has a driver node. If a Spot driver is reclaimed the whole run fails, so the estimate keeps one instance per cluster on demand.

## Related

* [Move Databricks Compute to Graviton](https://help.nops.io/inform/supported-recommendations/move-databricks-compute-to-graviton)
* [Run EMR Task Nodes on Spot Instances](https://help.nops.io/inform/supported-recommendations/run-emr-task-nodes-on-spot-instances)
* [Use Spot Node Pools for Non-Production GKE Clusters](https://help.nops.io/inform/supported-recommendations/use-spot-node-pools-for-non-production-gke-clusters)
* [Supported recommendations](https://help.nops.io/inform/supported-recommendations.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.