Hi team,
My view has ~100M records and I have a requirement to read it in Java code and process it. I'm using the below code
TableId destinationTableId = TableId.of("my_project","my_dataset","destination_table");
String query = "select col1, col2,..coln from my_project.my_dataset.my_view";
QueryJobConfiguration.Builder builder = QueryJobConfiguration.newBuilder(query)
.setUseLegacySql(false)
.setDestinationTable(destinationTableId)
.setCreateSession(false)
.setConnectionProperties(
List.of(
(ConnectionProperty.newBuilder()
.setKey("session_id")
.setValue("prev_session_id") // We save session_id from a previous query. It'll be just few minutes old
.build())));
JobId jobId = JobId.newBuilder().setLocation("US").setJob(UUID.randomUUID().toString()).build(); // My dataset is in US multi-region
Job queryJob = bigQuery.create(JobInfo.newBuilder(builder.build()).setJobId(jobId).build());
TableResult tableResult;
try {
queryJob = queryJob.waitFor();
if (queryJob == null) {
throw new RuntimeException("Job no longer exists");
}
tableResult = queryJob.getQueryResults();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException(e);
} catch(BigQueryException exp ){
throw exp;
}
for (FieldValueList record : tableResult.iterateAll()) {
// process record
}
The query succeeds and returns table result but I'm seeing com.google.cloud.bigquery.BigQueryException: Connection has closed: javax.net.ssl.SSLException: Connection reset while iterating through all the records. This error occurs sometimes after 3 hours, sometimes after 6 hours and sometimes within an hour.
What should I do to resolve the issue?
Thanks
Artifacts:
com.google.cloud:google-cloud-bigquery:2.5.1
Java 17