Skip to main content
GKE node pools can run on Spot VMs, which Compute Engine lists at 60% to 91% below on-demand but can reclaim at any time. Non-production clusters can usually handle that, because Kubernetes reschedules pods when a node goes away. nOps finds on-demand node pools whose cluster, node pool, node or project names mark them as non-production, and estimates the saving from moving them to Spot.

Why it matters

Development and test clusters often run on the same on-demand nodes as production. Spot node pools are one of the largest single savings available for those clusters.

How nOps finds it

nOps reads on-demand Compute Engine vCPU and memory billing for GKE nodes, grouped by node pool, over the last 7 complete days. A node pool is flagged when it was billed on all 7 days and at least 90% of its cost carries a non-production word (such as dev, test, staging or sandbox) in its cluster, node pool, node or project name, with no production word. Pools with more than 10% GPU or sole-tenant cost, pools already mostly on Spot, and pools where committed-use discounts cover more than half the cost are left out. The saving assumes Spot costs 40% of the on-demand list price (the low end of Google’s published 60-91% discount): it is the pool’s current cost after credits minus 40% of its list cost, scaled to a month, and must be at least $25 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.

Before you start

Workloads in the cluster must tolerate a node being removed at short notice (several replicas, or jobs that can restart).

How to implement

  1. Check which workloads run in the node pool and that they can be restarted on another node.
  2. Create a Spot node pool with the same machine type: gcloud container node-pools create POOL_NAME --cluster=CLUSTER --spot --machine-type=MACHINE_TYPE --region=REGION (or --zone).
  3. Move workloads to it, using a node selector on cloud.google.com/gke-spot=true or a taint and toleration.
  4. Keep a small on-demand node pool for system pods and anything that must not be interrupted.
  5. Scale down and delete the old on-demand pool, and update Terraform so it is not recreated.

Risks and considerations

  • Spot capacity can be reclaimed at any time with about 30 seconds’ notice, and may not be available in every zone at every moment.
  • The real discount depends on the machine type and region, so the saving can be higher than the estimate.
  • To roll back, move the workloads back to an on-demand node pool.

FAQs

From words in the cluster, node pool, node and project names only, such as dev, test, staging, qa or sandbox. Labels are not used for this recommendation.