Pagination issue with integrations

[FelipePelicer]

Folks, I'm facing a challenge with the integrations that we haven't been able to solve yet. The issue is related to pagination. Whenever we perform paginated searches, the last generated token returns a dataset smaller than expected. For example, if we have 682 rows in the table and set the value of ListEntitiesPageSize to 300, the second and last pagination token only brings back 80 rows instead of the expected 82. Additionally, if I set the pagination to a value larger than the number of results I expect to obtain, it works correctly. So, the problem lies within the pagination itself

[FelipePelicer]

Up

[Marramirez · Accepted solution]

Hello @FelipePelicer,

Welcome to Google Cloud Community

A reason that this is happening is because there is a data processing limit of 10MB during execution. If the cumulative size of all the integration data is greater than the limit, this might lead to reduced throughput, task failures and increased latencies.

You can check out these related GCC posts regarding pagination in Application Integration:

• API Pagination in Application Integration - Best Practice?
• Application Integration - mysql connector task

If the above options don't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[AndriiKorkoshko]

Hi everyone,

I encountered the same issue today while working with the Salesforce connector. My result set contained 21 items with a page size of 1, but I consistently received only 15 out of 21 items. This issue reproduces reliably, and the number of returned results remains the same each time.

I followed the API Pagination Best Practices mentioned above, but the issue persists. Has anyone identified a workaround or a solution to ensure all records are retrieved correctly? Any insights would be greatly appreciated.

Thank you!

[ssvaidyanathan]

@AndriiKorkoshko - Can you try paginating it using a smaller size?

[AndriiKorkoshko]

Like my page size is equal to 1 and on dataset of 21 items I am getting only 15
And it's constant behavior 

[ssvaidyanathan]

Hmm... not sure why thats the case. Did you try reaching out to Support?