Why it matters
Notebook instances are often started for an experiment and never stopped. One left running nights and weekends costs close to three times what it would on a working-hours schedule.How nOps finds it
nOps reads the last 30 complete days of SageMaker notebook instance billing. An instance is flagged when it was billed on average at least 20 hours a day, with at least 24 days of 20 or more hours, it has been billed since the start of the window and is still running. Instances mostly covered by Savings Plans are left out. The saving is the last 7 days of on-demand cost scaled to a month, times the share of the week outside 60 working hours (108 of 168 hours, about 64%). Savings under $5 a month are not shown.Before you start
Permission to update notebook instances and create lifecycle configurations (sagemaker:UpdateNotebookInstance, sagemaker:CreateNotebookInstanceLifecycleConfig).
How to implement
- Ask the notebook owner whether anything runs in it overnight, such as training in the notebook kernel.
- Create a lifecycle configuration that runs the AWS auto-stop-idle script, which stops the instance after a set idle time (SageMaker console: Notebook → Lifecycle configurations → Create configuration).
- Stop the notebook instance, attach the lifecycle configuration (Edit → Lifecycle configuration), and start it again.
- If you prefer a fixed schedule, call
aws sagemaker stop-notebook-instancefrom an EventBridge Scheduler rule in the evening. - If the instance is defined in Terraform or CloudFormation, add the lifecycle configuration there too.
Risks and considerations
- Attaching a lifecycle configuration needs a stop and start of the instance.
- Work in progress that is not saved to the instance volume is lost when it stops. The ML storage volume is kept and still billed.
- The estimate assumes a 60-hour working week. Teams in several time zones may need a longer schedule.
- To roll back, detach the lifecycle configuration or remove the schedule.
FAQs
Is my data deleted when the instance stops?
Is my data deleted when the instance stops?
No. Files on the instance’s ML storage volume are kept. Only the compute stops being billed.
Why 60 hours a week?
Why 60 hours a week?
It stands for 12 hours on each weekday. If your team works longer hours, the saving is lower.