[CloudRun] Long Delay on Initial Start & Execution...

ShaonDeyTC · 12-09-2024 05:23 AM

Description:

I have an application written with Python3. The used package list (requirements.txt) & Dockerfile can be found below.

requirements.txt

asyncpg==0.29.0
colorlog==6.8.2
httpx==0.27.2
jsonpickle==3.2.2
marshmallow-jsonapi==0.24.0
nats-py==2.9.0
pydantic==2.9.2
pydantic-settings==2.6.1
python-dotenv==1.0.1
SQLAlchemy==2.0.31
taskiq-faststream==0.2.0

Dockerfile

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV PORT=7000

WORKDIR /tc_core_shoppingbasket_backgroundservice

COPY . /tc_core_shoppingbasket_backgroundservice
RUN pip install --no-cache-dir --upgrade -r requirements.txt

EXPOSE $PORT
CMD ["sh", "-c", "taskiq scheduler main:app & python3 -m http.server -d /healthiness $PORT"]

This is a background service application that contains two types of services:

Subscriber which is Event-Triggered: Triggered when the subject on the messaging system (NATS JetStream) has data
Scheduler which is Time-Triggered: Triggered by a cron expression

Expectation: Using the same mentioned Dockerfile the built image is running as expected without any unexpected delays on both: Locally (Docker Desktop) & Cloud (AWS ECS). Want the similar response from GCP > Cloud Run > Service.

ISSUE: Whenever tried to run on GCP > Cloud Run > Service, facing two things:

VERY long delay on the initial start of the application instance (FastStream)
Even if the subject (on NATS) has data (not just one but multiple), takes a VERY long time for the Subscriber/Event-Triggered to pick it and process it

ronnelg

Hi @ShaonDeyTC,

Are you getting any error messages? Also can you share to us the logs from the Cloud Run? This will help us troubleshoot your issue. For the meantime, you may try the following and see if it fixes the issue:

Create a new Cloud Run service and deploy it there.
Reduce the size of your Docker image.
Increase your Cloud Run memory or CPU limit.
Change the execution environment of your service in Cloud Run from default to first or second generation, this will improve the CPI and network performance.
Optimize your Python applications for Cloud Run such as using smaller Python base images.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

ShaonDeyTC

Hi, @ronnelg, Thank you for the suggestions.

TLDR: Later, I'll share the logs from GCP. I'll try it running with a different Execution environment.

Details:

0. I'm not getting any error messages that denote this delay or application startup error. I'll try to export the logs from GCP & share it with you.

I tried several times to remove the older Cloud Run Service & create a new one. Each time, the same issue.
As you can see in the Dockerfile mentioned above, I believe it has been written optimally. Using that Dockerfile, the Docker image size I'm getting is currently ~187 MB. [PS: If you think this is larger, I am already running an API project (Python3, FastAPI, Uvicorn, Gunicorn) size of ~244MB running on GCP Cloud Run Service with Memory: 1 GiB, CPU: 1, Execution environment: Default. And it's running smooth as silk!]
Once, I felt it might be an issue with the GCP runtime configurations, especially Memory and CPU. So I even tried with Memory: 32 GiB and CPU: 8. But, the same issue.
Hmmm! This is something I have not tried at all. I kept it Default all the time. Will try it out for sure.
As you can see in the Dockerfile mentioned above, I'm already using a slim image of Python.

ShaonDeyTC

Here are the logs:

horizon_

I'm facing the same issue. were you able to resolve it?

ShaonDeyTC

No! I've switched to AWS ECS for this project and getting the results as expected. It's been 5 months on an issue being reported and not been even acknowledged properly from the support team!

@horizon_ are you facing the issue with the same specifications? Python, FastStream, NATS JetStream, GCP Cloud Run?

[CloudRun] Long Delay on Initial Start & Execution Time