Why it matters
Development and test pipelines are often copied from production with the standard execution class, although nobody needs them to start the moment they are scheduled.How nOps finds it
nOps reads the last 7 complete days of Glue job-run billing for standard ETL DPU-hours. A job is flagged when at least 90% of its cost is non-production, judged from its environment tag, then the environment nOps assigns, then words in the job name (dev, test, staging, QA, sandbox, UAT, demo and similar). Jobs billed at least 22 hours on every one of the 7 days are treated as streaming and left out. The saving is the job’s cost times the difference between the Flex and standard list prices for the region. Savings under $10 a month are not shown.Before you start
A Glue 3.0 or later Spark ETL job (not Python shell or streaming), and permission to update it (glue:UpdateJob).
How to implement
- Confirm the job is a Spark ETL job on Glue 3.0 or later and has no strict completion deadline.
- In the Glue console, open the job, go to Job details, and set Flex execution on, or pass
--execution-class FLEXtoaws glue start-job-runor setExecutionClasson the job. - Run it once and check the run time is acceptable.
- Update the job definition in Terraform, CloudFormation or your deployment scripts.
Risks and considerations
- Flex runs can start later and run longer, and a run can be interrupted and retried.
- Do not use Flex for jobs that feed time-sensitive production processes.
- To roll back, set the execution class back to standard.
FAQs
How does nOps know a job is non-production?
How does nOps know a job is non-production?
From its environment tag first, then the environment nOps assigns to the account, then words such as dev, test or staging in the job name.