Issue with BigQuery Data Streaming using google-cloud-bigquerystorage 2.24.2

[sanjaypratap25]

I am experiencing an issue with my BigQuery data streaming application, which uses the following dependency:

<dependency>

<groupId>com.google.cloud</groupId>

<artifactId>google-cloud-bigquerystorage</artifactId>

<version>2.24.2</version>

</dependency>

Issue Details:

1. Action Performed: The application uses JsonStreamWriter and BigQueryWriteClient to stream data to BigQuery. On 25/07/2024, it tried streaming data to bigquery.

2. Error Encountered: On 29/07/2024, we received an exception related to this operation.

3. Delay Concern:

   • We are concerned about the five-day delay between the action and the error occurrence.
   • We need to understand if the Google client library was retrying the operation on its end and why it took such a long duration.

Request for Support:

1. Retry Mechanism:

   • Please confirm if the Google Cloud BigQuery Storage client library has built-in retry mechanisms.
   • If so, could you provide details on the default retry settings and how we can configure custom retry settings to better handle such scenarios?

2. Issue Diagnosis:

   • Could you help us diagnose why there was a five-day delay before the exception was reported?
   • We need to understand if this is a known issue with version 2.24.2 of the client library and whether upgrading to a newer version is recommended. 

[jangemmar]

Hi @sanjaypratap25,

Welcome to Google Cloud Community!

Seems like you have a duplicate case here.

As answered by @ms4446, Yes, the google-cloud-bigquerystorage Java client library incorporates robust retry mechanisms. It automatically handles transient errors like network issues, temporary BigQuery service unavailability, and flow control signals. I highly encourage you to utilize the newer version as it introduces the timeout for in-flight requests.

I hope the above information is helpful.