Pagination issue with integrations

[FelipePelicer]

Folks, I'm facing a challenge with the integrations that we haven't been able to solve yet. The issue is related to pagination. Whenever we perform paginated searches, the last generated token returns a dataset smaller than expected. For example, if we have 682 rows in the table and set the value of ListEntitiesPageSize to 300, the second and last pagination token only brings back 80 rows instead of the expected 82. Additionally, if I set the pagination to a value larger than the number of results I expect to obtain, it works correctly. So, the problem lies within the pagination itself

[FelipePelicer]

Up

[Marramirez · Accepted solution]

Hello @FelipePelicer,

Welcome to Google Cloud Community

A reason that this is happening is because there is a data processing limit of 10MB during execution. If the cumulative size of all the integration data is greater than the limit, this might lead to reduced throughput, task failures and increased latencies.

You can check out these related GCC posts regarding pagination in Application Integration:

• API Pagination in Application Integration - Best Practice?
• Application Integration - mysql connector task

If the above options don't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!