Why it matters
Development and test clusters often run on the same on-demand nodes as production. Spot node pools are one of the largest single savings available for those clusters.How nOps finds it
nOps reads on-demand Compute Engine vCPU and memory billing for GKE nodes, grouped by node pool, over the last 7 complete days. A node pool is flagged when it was billed on all 7 days and at least 90% of its cost carries a non-production word (such as dev, test, staging or sandbox) in its cluster, node pool, node or project name, with no production word. Pools with more than 10% GPU or sole-tenant cost, pools already mostly on Spot, and pools where committed-use discounts cover more than half the cost are left out. The saving assumes Spot costs 40% of the on-demand list price (the low end of Google’s published 60-91% discount): it is the pool’s current cost after credits minus 40% of its list cost, scaled to a month, and must be at least $25 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.Before you start
Workloads in the cluster must tolerate a node being removed at short notice (several replicas, or jobs that can restart).How to implement
- Check which workloads run in the node pool and that they can be restarted on another node.
- Create a Spot node pool with the same machine type:
gcloud container node-pools create POOL_NAME --cluster=CLUSTER --spot --machine-type=MACHINE_TYPE --region=REGION(or--zone). - Move workloads to it, using a node selector on
cloud.google.com/gke-spot=trueor a taint and toleration. - Keep a small on-demand node pool for system pods and anything that must not be interrupted.
- Scale down and delete the old on-demand pool, and update Terraform so it is not recreated.
Risks and considerations
- Spot capacity can be reclaimed at any time with about 30 seconds’ notice, and may not be available in every zone at every moment.
- The real discount depends on the machine type and region, so the saving can be higher than the estimate.
- To roll back, move the workloads back to an on-demand node pool.
FAQs
How does nOps decide a cluster is non-production?
How does nOps decide a cluster is non-production?
From words in the cluster, node pool, node and project names only, such as dev, test, staging, qa or sandbox. Labels are not used for this recommendation.