> ## Documentation Index
> Fetch the complete documentation index at: https://help.nops.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Use FlexRS for Delay-Tolerant Dataflow Batch Jobs

> Run recurring batch Dataflow pipelines with FlexRS, which bills worker vCPU and memory 40% below the batch price.

Flexible Resource Scheduling (FlexRS) is a Dataflow option for batch jobs that do not need to start right away. Dataflow may delay the start of a FlexRS job, and in return bills its worker vCPU and memory at a lower rate. nOps finds batch pipelines that run regularly but only a few times a day, and estimates the saving from running them with FlexRS.

| | |
| - | - |
| **Cloud** | Google Cloud |
| **Category** | Analytics |
| **Service** | Dataflow |
| **Action** | Modify |
| **Effort** | Low |
| **Restart needed** | No |
| **Can be rolled back** | Yes |
| **Analysis window** | Last 14 days |
| **Source** | nOps |

## Why it matters

Many nightly and hourly-ish batch jobs only need to finish by a deadline, not start at an exact minute. For those, FlexRS is a setting change that cuts most of the job's compute cost.

## How nOps finds it

nOps groups Dataflow jobs into pipelines by project, region and job name (ignoring the launch timestamp in the name). A pipeline is flagged when it ran batch jobs on at least 3 of the last 14 days, ran a batch job in the last 2 days, has no streaming jobs, and averaged no more than 4 jobs a day over the last 7 days. The saving is 40% of the pipeline's batch vCPU and memory cost over the last 7 days, scaled to a month (FlexRS lists at $0.0336 against $0.056 per batch vCPU-hour). Disk and shuffle charges are not counted. The saving must be at least \$5 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.

## Before you start

Know how soon each run's output is needed. FlexRS jobs can start up to 6 hours after you submit them.

## How to implement

1. Confirm the pipeline's output is not needed within 6 hours of submission.
2. Add the FlexRS option when you launch the job: `--flexRSGoal=COST_OPTIMIZED` (Java) or `--flexrs_goal=COST_OPTIMIZED` (Python). For Dataflow templates, set the same option in the launch parameters.
3. Update the scheduler or orchestration (Cloud Composer, Cloud Scheduler, Workflows) that submits the job, and move its start time earlier if it has a deadline.
4. Check the next runs in the Dataflow console: the job shows as queued until FlexRS starts it.

## Risks and considerations

* A FlexRS job can wait up to 6 hours before it starts, so jobs with a tight deadline may finish late.
* FlexRS applies to batch pipelines only.
* To roll back, remove the FlexRS option from the launch command.

## FAQs

> [accordion] Why are pipelines that run many times a day left out?
>
> A pipeline that runs more than 4 times a day usually needs its results quickly, and a start delay of up to 6 hours would not fit.

## Related

* [Use BigQuery Storage Write API Instead of Streaming Inserts](https://help.nops.io/inform/supported-recommendations/use-bigquery-storage-write-api-instead-of-streaming-inserts)
* [Supported recommendations](https://help.nops.io/inform/supported-recommendations.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.