Shopify Integration List Product To BigQuery

[akashs]

I have created an API in the Shopify Store and created the Shopify Connection in GCP. It is an active and tested connection. 

In Application Integration, I have created primary integration and sub-integration in order to load the data to the BigQuery dataset. I cannot find instructions on Google's Gudies for listing Shopify Products to Bigquery so have looked at forums and tried to follow what others have done.. 

I am getting the below error, 

"'$[0].Id: string found, integer expected $[1].Id: string found, integer expected $[2].Id: string found, integer expected $[3].Id: string found, integer expected $[4].Id: string found, integer expected $[5].Id: string found, integer expected $[6].Id: string found, integer expected $[7].Id: string found, integer expected $[8].Id: string found, integer expected $[9].Id: string found, integer expected $[10].Id: string found, integer expected $[11].Id: string found, integer expected $[12].Id: string found, integer expected $[13].Id: string found, integer expected $[14].Id: string found, integer expected $[15].Id: string found, integer expected $[16].Id: string found, integer expected $[17].Id: string found, integer expected $[18].Id: string found, integer expected $[19].Id: string found, integer expected $[20].Id: string found, integer expected $[21].Id: string found, integer expected $[22].Id: string found, integer expected $[23].Id: string found, integer expected $[24].Id: string found, integer expected '"

[akashs]

I have now managed to create the sub-integration and main integration. I have tested the subintegration by entering the default value, which does work - i.e data is added to the BQ table.
But when I run the main integration, I get the below error:

 Sub-integration: - Required parameter Connector input payload is not present.
Main - integration:- 'Required parameter Connector input payload is not present.' eventMessages without params: '[name: "subintegration" event_info_id: " " trigger_id: "api_trigger/subintegration_API_1" client_id: " " execution_state: FAILED cloud_logging_details { } event_bus_status: ACTIVE snapshot_number: 12 ]' error/code: 'common_error_code: INVALID_PARAMETER'

i am nearly there to get this working... any help or direction will be appreciated.  

[akashs]

I found the issue and corrected it. I now have a list of product variants from Shopify in BigQuery! But created 28 rows, I would like the entire data. How can I do that? 

[prateekcholkar]

Hello @akashs - Could you please elaborate, how did you fixed this error. I am also facing the same issue and would like to understand

[akashs]

I have ended up using Fivetran connector, which is expensive, but you also
get more.

--
*This email may contain confidential and/or privileged
material/information. It is for the use of the named recipient(s) only. Any
review, retransmission, dissemination or other use of, or taking of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you have
received this mail by error, please notify the sender immediately and
delete this message from any computer.*

[gowthamnagarajn]

In Application Integration, when the connector task is used to list the products from Shopify, you can explicitly mention the pageSize (the number of results should be displayed per page). The default value for page size is 25. And, if it contains more than the specified pageSize, it returns a next page token (listEntitiesNextPageToken).

Workflow should be created in such a way that it checks the next page token and invokes the connector task again to get all the products in shopify.  

[akashs]

Thank you. I have increased the default limit (25) to the maximum limit
(200). But I get the below error: "Execution timeout, cancelled graph
execution. Please try again". The data could have rows more than 50k.
Should I increase the execution time?
I want to initially download all current data and then run a schedule run
at midnight for updates and new entries.
Is there a best practice guide that I can read to improve the workflow and
create a schedule run?

--
*This email may contain confidential and/or privileged
material/information. It is for the use of the named recipient(s) only. Any
review, retransmission, dissemination or other use of, or taking of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you have
received this mail by error, please notify the sender immediately and
delete this message from any computer.*

[gowthamnagarajn]

The default timeout for each connector task is 3 minutes. So, you can increase the Application Integration Workflow execution timeout. But again, the requests may timeout for 50k results within 3 mins. If it's timeout even after increasing the timeout, you can limit the results to 25, push it to BigQuery and use While loop task to iterate, query next page results and repeat.

[Snoshone07]

Hi @akashs it looks like you're running into a data type mismatch when loading Shopify data into BigQuery through Application Integration. Specifically, BigQuery expects the ID field as an integer, but the API is returning it as a string.

Possible Solutions:

Convert the field in BigQuery

• If the data is already in a staging table, you can convert the field using:

SELECT CAST(Id AS INT64) AS Id,*

FROM `your_dataset.your_table`

This will ensure the ID field is stored as an integer.

Adjust the schema in Application Integration

• Check if your integration allows you to explicitly define the data type before sending the data to BigQuery.

Use an Alternative Shopify-BigQuery Connector (Easier Setup)

• If the issue persists, you might consider a third-party solution like Windsor.ai, which directly integrates Shopify with BigQuery and automatically handles schema mismatches.
• This could be a faster and more reliable alternative to manually troubleshooting the GCP setup.

Hope this helps!