How to build REST API on Google Cloud?

[zeljana]

Hello community,

Do you know if there's a way to build REST API on GCP?

I need to build it in order to get data from outside of cloud source (SAP HANA) and store in BigQuery table. Using any integration tool (Fivetran, SAP Connectors, etc) is not an option since the client already has their own integration platform.

So, I'm wondering if there's a way to build API on google.

Thank you in advance!

[aakashsharmaa5]

Hi @zeljana 

Yes, you can build a REST API on Google Cloud Platform (GCP) to fetch data from SAP HANA and store it in BigQuery. Here's a high-level overview of how you can achieve this:

a. Enable the necessary APIs: Cloud Functions, BigQuery API, and Cloud Storage API

b. Use Google Cloud Functions to create a serverless REST API. Cloud Functions can be triggered by HTTP requests, making them suitable for building RESTful services

c. Use a Python library like pyhdb or hdbcli to connect to SAP HANA and execute SQL queries to fetch the required data

d. Use the BigQuery client library for Python to insert the fetched data into a BigQuery table

Once you deploy the cloud function you can invoke in through http request or cloud scheduler

Hope this helps.

thanks,
Aakash

[zeljana]

Hi Aakash,

So I went with a solution like this: Flask app deployed on Cloud Run that imports data into BQ table. Github action for unit testing of the app.

Right now I'm facing an issue with performance since the Cloud Run acts weird when uploading batch data with more then 1000 rows.

But all in all it worked with python and Flask prior to the testing of a batch (which is suppose to be on production) import of a lot of rows (~50MB).

Thank you for your reply though.

[aakashsharmaa5]

Hi @zeljana  good to hear on the progress. Did you check the metrics for memory usage and performance for the cloud run service. Also, in case you haven't already you can experiment with smaller batch size and change in memory, cpu and #instances of cloud run(scaling)

Explicitly triggering python garbage collection and use of asynchronous programming are some other options that may help to address performance issues, but would need some experimentation and monitoring. All the best !

thanks,
Aakash

[zeljana · Accepted solution]

Hi Aakash,

so the issue is that the Cloud Run has 32MB limit on HTTPS request and as I understand there's no way of changing that limit. But yes, I've tried setting up CR service with more memory and CPU but the issue wasn't with processing but the data amount that was in the HTTP request. 

Anyways, I've moved one from this solution to other services (Cloud Functions, BQ Load statement and Cloud Storage) but thank you for replying and giving me some ideas.

I'll close this convo now. Thanks again!