Skip to main content
BigQuery’s legacy streaming insert method (tabledata.insertAll) lists at $0.05 per GiB. The BigQuery Storage Write API ingests the same data for $0.025 per GiB, and the first 2 TiB each month per billing account are free. nOps finds projects that pay for legacy streaming inserts every day and estimates the saving from moving those writers to the Storage Write API.

Why it matters

Streaming ingestion runs all day, so its cost adds up quietly. The Write API is the method Google now recommends, and for many projects the free tier alone covers most of the volume.

How nOps finds it

nOps reads BigQuery streaming insert charges for each project and location over the last 14 complete days. A project is flagged when it was billed on at least 10 of those days. The volume is worked out from the cost at the $0.05 per GiB list price. The Write API cost is that volume at $0.025 per GiB, less a share of the 2 TiB monthly free tier split across the billing account’s projects by volume. No free tier is counted when the billing account already uses the Write API. The saving is the current monthly cost minus that Write API cost, and must be at least $5 a month. Projects whose spend on the service is at least 90% paid by credits or promotions are left out.

Before you start

Access to the code or pipeline that writes to BigQuery, and a test dataset to try the change on.

How to implement

  1. Find the jobs and services that stream into the project. Look for calls to insertAll or insert_rows in your code.
  2. Switch them to the Storage Write API. In the BigQuery client libraries, use the Storage Write API client (default stream for at-least-once, committed streams for exactly-once).
  3. For Dataflow or Apache Beam pipelines, set the BigQuery write method to STORAGE_WRITE_API (or STORAGE_API_AT_LEAST_ONCE).
  4. Test on a non-production dataset, then roll out writer by writer.
  5. After a few days, check that the streaming insert charges for the project have stopped.

Risks and considerations

  • This is a code change. Error handling, batching and retry behavior differ between the two APIs, so test before switching.
  • The 2 TiB free tier is shared by every project on the billing account, so the saving per project can be lower if other projects also move.
  • To roll back, switch the writer back to insertAll.

FAQs

No. The Storage Write API is also a streaming method, and data written to the default stream is available for queries right away.
Because the first 2 TiB a month per billing account is free on the Write API. Where that covers the project’s volume, most of the cost goes away.