API Pagination in Application Integration - Best Practice?

[piyushbhandari]

Hey team, 

I am using application integration service on Google Cloud to build automated workflows for business. I wanted to leverage application integration to build certain integrations with APIs that have paginated API response.

I could not find any documentation on doing so in Application Integration. Can you help me in figuring out less complicated solution to cycle through pages in the API response and store JSON response in an object?

[Meenchou · Accepted solution]

Hi Piyush 

For fetching paginated API response, you can use While loop task and subintegration containing your connector task

Parent integration

Your parent integration will have an Api trigger and while loop task that will call the subintegration in multiple iterations as long as the connector output payload of last iteration is not empty

Create two local variables in your parent integration -

connectorOutput: (type)Json var, (initial value) { }

This variable is referred in the while task condition (see screenshot). Initial value is {} (so that the condition is non falsy) for the first iteration. This variable is then overriden in While Loop task to hold subintegration `Task_1_connectorOutputPayload` 

When all the records are fetched Task_1_connectorOutputPayload and hence connectorOutput will be empty breaking the While loop 

pageToken: (type)String var, (initial value) check empty string radio button

This variable is used to get paginated response as it stores the  subintegration variable `Task_1_listEntitiesNextPageToken` of last iteration and passes it to the next iteration

Sub integration

Sub integration will contain an Api trigger and connector task 

Pls note the following points for the input and output of connector task

Task_1_listEntitiesPageSize : Provide any default value (ex 100)

Task_1_listEntitiesNextPageToken: Set it as input (to be overriden by parent integration) and output of the integration (to be collected by parent integration)

Task_1_listEntitiesSortByColumns: provide the primary key of the table here (or any other field) 

Task_1_connectorOutputPayload: Set it as output of the integration (to be collected by parent integration)

Parent integrationSub-integration