Application Integration - BigQuery connector - Pagination

[padmanabhamv]

I'm using application integration to retrieve BigQuery records in pages. My table has 100+ records, and I'm fetching them 5 at a time. On the first call, I set listEntitiesPageSize to 5 and leave listEntitiesPageToken empty. This successfully retrieves the first 5 records, but the listEntitiesNextPageToken in the output is empty. How can I retrieve the next 5 records (second page) in this scenario? When more records are there why listEntitiesNextPageToken came as empty?

[kanavgupta]

Hi @padmanabhamv 

Bigquery connector does not have support for pagination as of now. You can set higher listEntitiesPageSize to retrieve all the records at once.

[padmanabhamv]

@kanavgupta  @shaaland  , Are you 100% sure and confirming from product side?

What is the purpose of these filters listEntitiesPageSize, listEntitiesPageToken, and listEntitiesNextPageToken ?

[kanavgupta]

You can still achieve pagination by mentioning the column name in the sortByColumn field. On mentioning it, we will use that column for indexing in pagination and pagination will work as usual giving you listEntitiesNextPageToken in response.

[padmanabhamv]

ok, with that, when i retrieve first few columns, i am getting details and listEntitiesNextPageToken is null, so not sure how to get a next page or next setup or records. 

[DMulchandani]

@padmanabhamv 

Could you please try again. It should work as expected. If it is not working please share the project id and connection name in direct message. we will look into it

[dejancvetkoski]

@padmanabhamv  did you find some solution? I have the same issue, I have > 500.000 records in BigQuer, created a connector, added (listEntitiesPageSize = 200) and implemented "While Loop" but inside the logs I have only 1 iteration and "listEntitiesNextPageToken" always is empty (null)... Appreciate your help,  thanks in advance!

cc: @DMulchandani @kanavgupta 

[AlexET]

Hi @dejancvetkoski, thanks for reaching out! The issue should now be resolved. Could you confirm if you have set the sortByColumn field, ensuring that it has unique values? This is necessary for pagination to work correctly.

If you’re still experiencing issues after verifying this, feel free to share more details, and we’ll look into it further.

[dejancvetkoski]

Hello @AlexET thanks for the response, yeah, it works with sortByColumn field added. Thanks again, cheers!