This page:
https://cloud.google.com/bigquery/quotas#write-api-limits
lists a 10MB limit on Storage Write API requests
| Request size | 10 MB | The maximum request size is 10 MB. |
this is quite low and we are wondering how to increase it? e.g., we get exceptions in our program:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: MessageSize is too large. Max allow: 10000000 Actual: 10003187
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at cdao.dpde.App.main(App.java:140)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:254)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: MessageSize is too large. Max allow: 10000000 Actual: 10003187
at cdao.dpde.WritePendingStream$DataWriter$AppendCompleteCallback.onFailure(WritePendingStream.java:178)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:761)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1105)
at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:62)
at cdao.dpde.WritePendingStream$DataWriter.append(WritePendingStream.java:132)
at cdao.dpde.WritePendingStream.writePendingStream(WritePendingStream.java:51)
at cdao.dpde.Consumer.run(Consumer.java:160)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: MessageSize is too large. Max allow: 10000000 Actual: 10003187
at com.google.cloud.bigquery.storage.v1.StreamWriter.appendInternal(StreamWriter.java:301)
at com.google.cloud.bigquery.storage.v1.StreamWriter.append(StreamWriter.java:292)
at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:153)
at cdao.dpde.WritePendingStream$DataWriter.append(WritePendingStream.java:131)
... 6 more
LinkedIn
Twitter
