Shopify Data to BigQuery - ForEach and Pagination

[dbmuser]

Hello,  

I'm very new to this tool, but I'm a little confused with how to use While Loop and a ForEach task together.

Shopify has a product entity and provides 25 products per page.  A next page token is provided.  In an app I'd call the foreach while the nextpage token is not empty.  

I've created an integration and a sub-integration.  I'm able to write the 25 products to bigQuery.  I just don't know how to start the process over for the next page.  

[dianjinli]

Hi, dbmuser,

When you add a while/foreach loop task, you can add condition for that, for example, "pageToken$pageToekn$ != "" " to check if the page token is empty now

Thanks,

Dianjin

[dbmuser]

I'm a little confused on how to set up.  Right now I have 2 integrations set up where it only processes the first payload of 25 products.  When I read about the WhileLoop, it seems I need to also call a sub integration.  Does that mean I need 3 integrations for this one task?  

Integration 1 - Parent
Trigger > Shopify connector > ForEachLoop > Calls Child Integration

Integration 2 - Child
Trigger > Data Mapping > BigQuery Connector

[akashs]

Hi 

Did you figure it out? I have the same issue, I have managed to create the integration but only 25 rows were added to the dataset. I need to get the entire list and then run a routine for new products that are being added. 

[lsolatorio]

Hi @dbmuser,

Welcome back and thank you for reaching out to our community.

You can check out this post from our forums. One of the answers has a similar use case as yours, utilizing foreach loop in a shopify integration, you may find it helpful.

you can use foreach loop task to loop through that array and call the subintegration which would eventually call big query connector and the output from the task can be collected in your parent integration

---