Why it matters
Streaming ingestion runs all day, so its cost adds up quietly. The Write API is the method Google now recommends, and for many projects the free tier alone covers most of the volume.How nOps finds it
nOps reads BigQuery streaming insert charges for each project and location over the last 14 complete days. A project is flagged when it was billed on at least 10 of those days. The volume is worked out from the cost at the $0.05 per GiB list price. The Write API cost is that volume at $0.025 per GiB, less a share of the 2 TiB monthly free tier split across the billing account’s projects by volume. No free tier is counted when the billing account already uses the Write API. The saving is the current monthly cost minus that Write API cost, and must be at least $5 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.Before you start
Access to the code or pipeline that writes to BigQuery, and a test dataset to try the change on.How to implement
- Find the jobs and services that stream into the project. Look for calls to
insertAllorinsert_rowsin your code. - Switch them to the Storage Write API. In the BigQuery client libraries, use the Storage Write API client (default stream for at-least-once, committed streams for exactly-once).
- For Dataflow or Apache Beam pipelines, set the BigQuery write method to
STORAGE_WRITE_API(orSTORAGE_API_AT_LEAST_ONCE). - Test on a non-production dataset, then roll out writer by writer.
- After a few days, check that the streaming insert charges for the project have stopped.
Risks and considerations
- This is a code change. Error handling, batching and retry behavior differ between the two APIs, so test before switching.
- The 2 TiB free tier is shared by every project on the billing account, so the saving per project can be lower if other projects also move.
- To roll back, switch the writer back to
insertAll.
FAQs
Will my data arrive later?
Will my data arrive later?
No. The Storage Write API is also a streaming method, and data written to the default stream is available for queries right away.
Why can the saving be more than half the cost?
Why can the saving be more than half the cost?
Because the first 2 TiB a month per billing account is free on the Write API. Where that covers the project’s volume, most of the cost goes away.