Why it matters
Many background workloads call Bedrock the same way an interactive app does and pay the on-demand price, even though nobody is waiting for the answer.How nOps finds it
nOps reads the last 7 complete days of Bedrock billing for on-demand, non-streaming model calls (streaming calls are treated as interactive and left out). The saving is the token cost times the difference between the batch and on-demand price for that model and region. The batch price comes from the AWS price list, or from your own billed batch rate when you already use batch. Models with no known batch price are not shown, and neither are savings under $25 a month. Because nOps cannot tell which calls can wait, this is the most you could save if all of them moved.Before you start
A model that supports batch inference in your region, an S3 bucket for input and output, and an IAM service role Bedrock can use to read and write it.How to implement
- Find which callers of this model do not need an immediate answer (for example nightly enrichment or evaluation jobs).
- Write their requests as JSONL records to an S3 bucket.
- Create a batch inference job (Bedrock console: Inference → Batch inference → Create job, or
aws bedrock create-model-invocation-job) pointing at the input and output locations. - Change the caller to read results from the output location when the job finishes.
- Keep interactive and streaming traffic on on-demand inference.
Risks and considerations
- Batch jobs complete asynchronously, so they are not suitable for anything a user is waiting on.
- Batch jobs have minimum and maximum record counts per job, so very small workloads may need to be grouped.
- The estimate covers every non-streaming call; the real saving depends on how much of that traffic can wait.
- To roll back, send the requests on demand again.
FAQs
Does nOps count my streaming calls?
Does nOps count my streaming calls?
No. Streaming calls are treated as interactive and left out of the estimate.
How long do batch jobs take?
How long do batch jobs take?
AWS aims to complete batch jobs within 24 hours. Most finish sooner, but plan for the full window.
Related
- Move Latency-Tolerant Azure OpenAI Workloads to Batch
- Move Latency-Tolerant Gemini Workloads to Batch or Flex
- AI Model Tier Right-Sizing
- Supported recommendations