Skip to main content
Flexible Resource Scheduling (FlexRS) is a Dataflow option for batch jobs that do not need to start right away. Dataflow may delay the start of a FlexRS job, and in return bills its worker vCPU and memory at a lower rate. nOps finds batch pipelines that run regularly but only a few times a day, and estimates the saving from running them with FlexRS.

Why it matters

Many nightly and hourly-ish batch jobs only need to finish by a deadline, not start at an exact minute. For those, FlexRS is a setting change that cuts most of the job’s compute cost.

How nOps finds it

nOps groups Dataflow jobs into pipelines by project, region and job name (ignoring the launch timestamp in the name). A pipeline is flagged when it ran batch jobs on at least 3 of the last 14 days, ran a batch job in the last 2 days, has no streaming jobs, and averaged no more than 4 jobs a day over the last 7 days. The saving is 40% of the pipeline’s batch vCPU and memory cost over the last 7 days, scaled to a month (FlexRS lists at $0.0336 against $0.056 per batch vCPU-hour). Disk and shuffle charges are not counted. The saving must be at least $5 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.

Before you start

Know how soon each run’s output is needed. FlexRS jobs can start up to 6 hours after you submit them.

How to implement

  1. Confirm the pipeline’s output is not needed within 6 hours of submission.
  2. Add the FlexRS option when you launch the job: --flexRSGoal=COST_OPTIMIZED (Java) or --flexrs_goal=COST_OPTIMIZED (Python). For Dataflow templates, set the same option in the launch parameters.
  3. Update the scheduler or orchestration (Cloud Composer, Cloud Scheduler, Workflows) that submits the job, and move its start time earlier if it has a deadline.
  4. Check the next runs in the Dataflow console: the job shows as queued until FlexRS starts it.

Risks and considerations

  • A FlexRS job can wait up to 6 hours before it starts, so jobs with a tight deadline may finish late.
  • FlexRS applies to batch pipelines only.
  • To roll back, remove the FlexRS option from the launch command.

FAQs

A pipeline that runs more than 4 times a day usually needs its results quickly, and a start delay of up to 6 hours would not fit.