Skip to main content
An all-purpose (interactive) cluster bills Databricks DBUs, and the cloud instances under it, for every hour it runs, whether or not anyone is using it. nOps finds clusters that run almost all the time and sit at their minimum size outside business hours, and estimates the saving from letting them terminate then.

Why it matters

An interactive cluster with auto-termination turned off, or kept awake by an attached notebook or scheduled job, costs as much overnight and at weekends as during the working day.

How nOps finds it

nOps reads the last 30 days of Databricks billing. A resource is flagged when it was billed in at least 90% of the hours of the last 30 days (or since your first Databricks day, if later), was still running on the last complete day, and sat at its minimum level (within 5% of its 10th-percentile hourly usage) in at least 80% of off-hours. Off-hours are weekends and weekday hours outside 13:00-24:00 UTC. The saving is what the cluster cost in those idle off-hours, scaled to 30 days, and must be at least $10 a month. When nOps can match the cluster’s EC2 instances in your AWS bill, their off-hours cost is included; otherwise the saving is DBUs only and the real saving is higher.

Before you start

Know who uses the cluster outside business hours, including attached notebooks, scheduled jobs and streams.

How to implement

  1. Check what runs on the cluster at night and at weekends: attached notebooks, scheduled jobs, streams or polling processes.
  2. In Databricks, open Compute, edit the cluster, and set Terminate after 30 to 60 minutes of inactivity.
  3. Move scheduled or polling work to job compute, which starts and stops with each run.
  4. Check after a week that the cluster now terminates outside business hours.
  5. If the cluster is defined in Terraform or a cluster policy, set autotermination_minutes there too.

Risks and considerations

  • Restarting a terminated cluster takes a few minutes, so the first notebook run of the day waits.
  • Work that genuinely runs overnight must move to job compute first, or it stops when the cluster terminates.
  • If the business hours used by nOps do not match your team’s, the saving may be lower.

FAQs

No. A terminated all-purpose cluster keeps its configuration and can be restarted at any time.