How to download data, then feed it into Magento?

[jterblanche]

Hi all,

Complete GCP newbie here - just discovered GCP a couple days ago, and completely overwhelmed by the sheer number of products, services and options available.

My requirement is fairly simple though (3 steps):

1.) Download a CSV or XML data feed from my wholesaler.

2.) Validate and Convert Data.

3.) Insert / Update Categories, Products and Stock in Magento.

I fully understand there is probably a million different ways to achieve this, just looking for some advice as to which GCP products are suited/designed to make this process easier to develop and maintain in future, considering that future versions/schemas of the source and destination datasets may change.

Thanks in advance

J

[dchiesa1]

You've posted your query in a forum that is dedicated to API Management, which is a fairly specific corner of the GCP ecosystem. Probably not the optimal place to begin your search. 

Even so I will try to give you some guidance.  

If I were a consultant, I would tell you: Your requirements aren't  well specified enough to allow someone to make a hard recommendation.  

The basic idea of "download some stuff, validate and maybe modify it, then upsert/merge it back to a remote datastore".... is a very general description.  You haven't specified:

• how often will this happen?  Once?  once per month? once per day?  thousands of times per second? 
• how will the job get triggered?  By opening an app on your desktop?  By an HTTP request sent to a remote system?  on a timer?  via some other trigger?
• how large is the data?  10kb?  10mb?  10TB? 
• how involved is "validate and convert"?  

I'm not an expert in Magento, so I don't have any good guesses as to the answers to the above. Depending on the answers, you may want to look at different things. 

If you need to run a job once per month, and it deals with 1mb of data, then I don't suppose you even need a cloud platform.  You don't need GCP to support that. You can build a program that runs on your local PC, using Java, .NET, python, JavaScript, or something else.... and it will do all that work. Trigger it with a click on your desktop, and it will do what you want. It might take 5 seconds to run. 

You might even be able to build that in Excel.

But, there might be a simpler way.  I think there are probably "higher level" tools that are  well suited to this sort of thing. You could look into those Like maybe Zapier. 

Disclaimer: I'm not affiliated with Zapier, and I have never used the Zapier+Magento integration. But it seems like it would be easy to use.

[henryjames]

For your needs, I suggest using Cloud Functions to automate downloading and processing the CSV/XML feed. Use Cloud Storage to store the data, and Cloud SQL or BigQuery for data validation and transformation. Finally, integrate with Magento’s API for inserting/updating products. I looked into this because I’m also working on optimizing product management for my online store.