Why it matters
An interactive cluster with auto-termination turned off, or kept awake by an attached notebook or scheduled job, costs as much overnight and at weekends as during the working day.How nOps finds it
nOps reads the last 30 days of Databricks billing. A resource is flagged when it was billed in at least 90% of the hours of the last 30 days (or since your first Databricks day, if later), was still running on the last complete day, and sat at its minimum level (within 5% of its 10th-percentile hourly usage) in at least 80% of off-hours. Off-hours are weekends and weekday hours outside 13:00-24:00 UTC. The saving is what the cluster cost in those idle off-hours, scaled to 30 days, and must be at least $10 a month. When nOps can match the cluster’s EC2 instances in your AWS bill, their off-hours cost is included; otherwise the saving is DBUs only and the real saving is higher.Before you start
Know who uses the cluster outside business hours, including attached notebooks, scheduled jobs and streams.How to implement
- Check what runs on the cluster at night and at weekends: attached notebooks, scheduled jobs, streams or polling processes.
- In Databricks, open Compute, edit the cluster, and set Terminate after 30 to 60 minutes of inactivity.
- Move scheduled or polling work to job compute, which starts and stops with each run.
- Check after a week that the cluster now terminates outside business hours.
- If the cluster is defined in Terraform or a cluster policy, set
autotermination_minutesthere too.
Risks and considerations
- Restarting a terminated cluster takes a few minutes, so the first notebook run of the day waits.
- Work that genuinely runs overnight must move to job compute first, or it stops when the cluster terminates.
- If the business hours used by nOps do not match your team’s, the saving may be lower.
FAQs
Does terminating a cluster lose its configuration?
Does terminating a cluster lose its configuration?
No. A terminated all-purpose cluster keeps its configuration and can be restarted at any time.
Related
- Stop Always-On SQL Warehouse
- Use Spot Workers for Non-Production Databricks Compute
- Schedule Always-On Databricks Streaming Jobs
- Supported recommendations