Why does Google App Engine 2nd generation is slower than the 1st generation?

[rafita96]

We had an App Engine's standard environment application running in Python 2.7, and we upgraded it to Python 3.11.

Services/frameworks used by old app:

• Shared Memcache
• Task Queue
• Google's Discovery library (for Big Query)
• webapp2 Framework

Services/frameworks used by new app:

• Flask
• Shared Memcache
• Task Queue
• Storage Write API
• webapp2 was removed since it isn't compatible with Python 3.

We followed this Google's guide to upgrade our application and use the legacy bundled services: https://cloud.google.com/appengine/docs/standard/python3/services/access#python-3

The codebase is simple. It reads a request and stores it in Big Query. It uses the Memcache service to throttle requests and the Task queue to call the "Store to Bigquery" endpoint in the same application.

After the upgrade we saw the following:

Higher latency: from ~50ms to ~250ms Higher number of instances: from 6 to 50 using the same workload

We tried to run the Google's profiler but the platform couldn't install the package for Python 3.11.

We created and deployed a "hello world" application to discard performance issues with Python 3.11, Flask, Memcache, Taskqueue, and BigQuery streaming insertion.

We found that the BigQuery legacy insertion was causing a latency delay of ~500ms. Then we implemented the Storage Write API but we ended with a latency of ~250ms, which isn't comparable with the previous performance.

Is this an issue with the App Engine platform? Are there another steps to avoid this kind of delays?

We expect the same or better performance as the previous version.

[Marramirez]

Hello @rafita96,

Welcome to the Google Cloud Community!

Take a look at this documentation about Troubleshooting Elevated Latency in your Application. 

You have to narrow down the root cause for the latency spikes. You have to first define the scope of the issue, each application is different in terms of its monitoring and logging, these will help in narrowing down the root cause of higher latency and determine what you can do to trouble shoot it.

To determine which request path is causing the latency or errors, you have to identify where it most likely will happen. The main components are:

1. Client
2. Internet
3. Google Front End (GFE)
4. App Engine Serving Infrastructure
5. Application Instance

There are scenarios that might cause the higher latency in your App Engine. You can check the given link as you might be encountering some of these scenarios.

You can also check Google Cloud Service Health Incidents. Here you will see the history of incidents reported by product. There has also been an incident in Google App Engine that users from multiple regions are observing latency issues.

If the above options don't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[rafita96]

Hi @Marramirez

Thanks for your reply. I read the Troubleshooting document already.

The mean latency increased by 865% for the endpoint where I store to BigQuery. I'm sure that the call to the BigQuery API cause the delay, even if I'm using the Storage Write API implementation.

Also, I noticed that my Python 3.11 version handles around 2 QPS per instance, meanwhile my previous Python 2.7 version handles up to 14 QPS per instance. About the Runtime MCycles metric, the 3.11 version shows the double than the 2.7 version.

Right now, I'm trying a Pub/Sub with BigQuery implementation but I'm running out of options.

[rafita96 · Accepted solution]

Hi everyone,

I got 15% less latency by moving out my code from Python 2.7 to Go 1.20.

Best regards.

[troberti]

We also faced this issue when upgrading our backends to Python 3. Somehow the Google BigQuery client is very slow with about 500ms per streaming insert. It probably does a lot of unnecessary requests. Making a direct HTTP requests using the BigQuery V2 REST  API is much faster, about <100ms. 

We made a small library that does this: https://github.com/firi/appengine-afterburner.