GCP Cloud Run Container - Updating Data Queries

[tangedahl]

I have a Dash App running in a GCP Cloud Run container.  This is a data dashboard that makes calls back to a GCP Postgres instance to update data that refreshes every 30 min.  If the app is being access often the I believe the container stays refreshed and allows the for the app to refresh quickly.  However when its is accessed often the container is not making the database calls and refreshing the data.  How can I periodically have the GCP Cloud Run container run functions in the code and query new data?

Thanks,
Eric

[Joy_S]

Hi @tangedahl,

It seems you're encountering a scenario where your Cloud Run container, designed to refresh data every 30 minutes, isn't consistently making database calls when the app isn't accessed frequently. This is likely due to Cloud Run's autoscaling behavior, which can scale down to zero instances when there's no traffic. To resolve this, you can implement a cron-like task scheduler that triggers certain functions at regular intervals, such as querying new data from your GCP Postgres instance every 30 minutes, even when the app is not actively being accessed. Here's how you can achieve that:

1. Use Google Cloud Scheduler and Cloud Pub/Sub - Since Cloud Run is a serverless service, it doesn't have a built-in mechanism to run background tasks or scheduled jobs. However, you can use Google Cloud Scheduler combined with Cloud Pub/Sub to send periodic triggers to your Cloud Run app to run specific background tasks, like refreshing the data.
   1. Set up Google Cloud Pub/Sub: Create a Pub/Sub topic that will be used to trigger your Cloud Run service then create a subscription for that topic.
   2. Modify Cloud Run App to Listen to Pub/Sub Messages: In your Cloud Run app, you need to add an HTTP endpoint that listens for incoming requests from Pub/Sub (which will be triggered every 30 minutes). This endpoint will handle the logic for fetching fresh data from the Postgres database.
   3. Use Google Cloud Scheduler to Trigger the Pub/Sub Topic: Create a Cloud Scheduler job that publishes a message to the Pub/Sub topic at a regular interval (e.g., every 30 minutes).
   4. Write Code to Handle Data Refresh: Inside your Cloud Run container, you should implement logic that gets triggered when the Pub/Sub message is received. This will allow the app to periodically refresh the data in the background, even if the app is not being accessed by users.
2. Use Cloud Functions: Instead of using Cloud Run to handle periodic tasks, you could set up a Google Cloud Function to query the data, and then have Cloud Scheduler trigger that function. Cloud Functions are often used for lightweight, scheduled tasks like this and might be more cost-effective for periodic jobs.

I hope this helps!

[tangedahl]

Ok I got most of this setup, first question however is don't I want a Pub/Sub endpoint, not an HTTP?

I have a topic created, I set a eventarc trigger in the Google Cloud run app setting which points to that topic, then I have a scheduler setup to to run that topic periodically.  

This is code running in my cloud run app:

@functions_framework.cloud_event

def hello_pubsub(cloud_event😞

    # Print out the data from Pub/Sub, to prove that it worked

    print(base64.b64decode(cloud_event.data["message"]["data"]))

    refresh_data()


The refresh data function just updated the dataframe:

def refresh_data():

    """

    Refreshes the df_rank DataFrame by querying the SQL database.

    """

    with pool.connect() as db_conn:

        df_rank = pd.read_sql_table(.....

Currently this doesn't seem to updated the data when I refresh the web app however.  Any insight would be helpful!

Thanks,

Eric