Why it matters
Databricks node types are usually picked once in a cluster or job template and reused everywhere, so one change can lower the cost of many runs.How nOps finds it
nOps reads the last 7 days of on-demand EC2 billing for instances tagged as launched by Databricks, grouped by job, SQL warehouse, instance pool or cluster and by instance type. Instances in the m5 to m7, r5 to r7 and c5 to c7 x86 families (including the ādā variants with local disk) are compared with the same size in the 6g and 7g Graviton families, and i3 with i4g. The saving is the on-demand EC2 cost times the list-price difference to the cheaper Graviton type in the same region. Databricks DBU charges are not included. Hours covered by Reserved Instances or Savings Plans and savings under $5 a month are left out.Before you start
A Databricks Runtime version that supports Graviton, and libraries or init scripts that work on Arm64.How to implement
- Check the cluster, job or pool runs a Databricks Runtime version that supports Graviton, and that its libraries and init scripts work on Arm64.
- Edit the compute and pick the recommended Graviton node type (for example m6gd or r7gd) for workers and the driver.
- For jobs, change the node type in the job cluster definition; for instance pools, create a pool with the Graviton type and point clusters at it.
- Run the job or a test query and compare run time.
- Update cluster policies, Terraform or Databricks Asset Bundles so new clusters use the Graviton type.
Risks and considerations
- Some features and libraries are not available on Graviton nodes. Test before switching production jobs.
- Run time can change, which changes DBU cost in either direction.
- To roll back, set the node type back to the x86 type.
FAQs
Does the saving include Databricks DBUs?
Does the saving include Databricks DBUs?
No. It covers the EC2 instances only. DBU rates for Graviton node types may differ.