Why it matters
Many nightly and hourly-ish batch jobs only need to finish by a deadline, not start at an exact minute. For those, FlexRS is a setting change that cuts most of the job’s compute cost.How nOps finds it
nOps groups Dataflow jobs into pipelines by project, region and job name (ignoring the launch timestamp in the name). A pipeline is flagged when it ran batch jobs on at least 3 of the last 14 days, ran a batch job in the last 2 days, has no streaming jobs, and averaged no more than 4 jobs a day over the last 7 days. The saving is 40% of the pipeline’s batch vCPU and memory cost over the last 7 days, scaled to a month (FlexRS lists at $0.0336 against $0.056 per batch vCPU-hour). Disk and shuffle charges are not counted. The saving must be at least $5 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.Before you start
Know how soon each run’s output is needed. FlexRS jobs can start up to 6 hours after you submit them.How to implement
- Confirm the pipeline’s output is not needed within 6 hours of submission.
- Add the FlexRS option when you launch the job:
--flexRSGoal=COST_OPTIMIZED(Java) or--flexrs_goal=COST_OPTIMIZED(Python). For Dataflow templates, set the same option in the launch parameters. - Update the scheduler or orchestration (Cloud Composer, Cloud Scheduler, Workflows) that submits the job, and move its start time earlier if it has a deadline.
- Check the next runs in the Dataflow console: the job shows as queued until FlexRS starts it.
Risks and considerations
- A FlexRS job can wait up to 6 hours before it starts, so jobs with a tight deadline may finish late.
- FlexRS applies to batch pipelines only.
- To roll back, remove the FlexRS option from the launch command.
FAQs
Why are pipelines that run many times a day left out?
Why are pipelines that run many times a day left out?
A pipeline that runs more than 4 times a day usually needs its results quickly, and a start delay of up to 6 hours would not fit.