Has anyone tried the new App Engine (dev_appserver) for Python 3 on Windows?

[micdearmas]

A few days ago was released a new version of Google Cloud CLI (468.0.0) that in the release notes says:

* Upgraded dev_appserver from Python 2 to Python 3.
...
* Python 2 is no longer supported and should not be needed in your environment.

I was very happy to see that, but when I go to the online documentation I found this:

However, the docs for Local development server options also say this:

so maybe it does work on Windows and the docs haven't been updated?

Has anyone tried it?

Also, in that last docs page, I don't find the flag --python_virtualenv_path, which I'm using now with an older version of the CLI. I don't know if that's just missing from the docs or it was removed and now that functionality has to be achieved in another way when using dev_appserver.

I'm just trying to gather as much info as possible before instaling anything new in my dev. env. because I rely on it to maintain several apps. I'm using (on Windows 10) Python 3.9 (and Python 2.7 because it was still required to use dev_appserver) and CLI 431.0.0. My plan was to install in the same PC (without removing anything that is currently working) Python 3.10 and CLI 468.0.0 and progressively start migrating to the new versions, so that I don't need Python2 any more.

Any suggestions/comments (other than forgetting about Windows 🙂 ) would be appreciated.
Thanks.

[robertcarlos]

Hi @micdearmas,

Welcome to Google Cloud Community!

There is a workaround that was published by NoCommandLine that you may want to try as dev_appserver tool doesn't work on Windows. Please check this link on using dev_appserver.py for Python 3 Apps on Windows.

You may also check the following links that is related to your question:

• Migrating App Engine apps from Python27 to Python3 on Windows 
• Python 3.7 Local Development Server Options for new app engine apps 

If the steps didn't work, you may file a feature request so that our engineers could take a look at this. We don't have a specific ETA once the ticket has been created.

Hope this helps.

[micdearmas]

Thank you @robertcarlos ,

I've been using the patch from @NoCommandLine for a while now and I was just wondering if this brand new version released just a few days ago wouldn't need the patch anymore. From your answer, I guess I should assume that it still doesn't work on Windows out of the box and that the existing patch still works with the new version of the CLI.

I'll see if I can make it work without breaking anything that is now working in my dev. env. and if I succeed (meaning that I don't need Python 2.7 anymore); then, I'll look into running things without the use dev_appserver, but still using all the functionality within google.appengine.api in Python 3 alone, as that's the other big question I have about this release.

Thanks again.

[NoCommandLine]

FYI -

Someone on gcloud SDK460.0.0 found out the patch didn't work on that version. They had to downgrade to a lower version of gcloud SDK to use the patch. 

I haven't had time to take a look at changes in newer version of gcloud SDK and if it has any impacts on the patch (and update the patch if necessary).

......NoCommandLine ......

 https://nocommandline.com

      Analytics & GUI for 
App Engine & Datastore Emulator

[micdearmas]

Thank you @NoCommandLine for the heads up!

I guess my getting rid of Python 2.7 will have to wait a little longer 😪

Is there a way to subscribe (to some kind of release notes) to get an email when a new release of the patch is made available?

[NoCommandLine]

I'll send you a message via this forum or you can look at Github subscriptions

.....NoCommandLine ......

 https://nocommandline.com

      Analytics & GUI for 
App Engine & Datastore Emulator

[micdearmas]

There was a new release yesterday with changes to devappserver. I'm not sure about the specifics of the change (other than changing the version number), but I thought it might affect this.

[NoCommandLine]

@micdearmas 

A new version of the Patch to support gcloud SDK v470.0.0 is now available. You can find it here.

There is a known issue and it's described here  Issue has been fixed.

.....NoCommandLine ......

 https://nocommandline.com

      Analytics & GUI for 
App Engine & Datastore Emulator

[micdearmas]

Terrific!

Thanks a lot @NoCommandLine; I'll make sure to contribute a tip later this week 🙂

[micdearmas · Accepted solution]

Just posting the solution here the way I got it:

1- I installed Python 3.11 without adding it to the Windows (env. var.)PATH, so whatever I had already setup and working on that PC stays the same.

2- Downloaded the desired new version (470.0.0) of the SDK/CLI (as a windows-x86_64.zip) from here.

3- Unzipped it to the same location that contains previous versions of the SDK/CLI just in a new folder for it ("C:\Program Files (x86)\Google\Cloud SDK 470")

4- Installed/updated components:

"C:\Program Files (x86)\Google\Cloud SDK 470\google-cloud-sdk\bin\gcloud" components install app-engine-python-extras

5- Got and installed the latest patch for Windows from @NoCommandLine  from here.

6- Modified the batch file that starts dev_appserver for each application that I'm upgrading.

<path_to_python311>\python.exe <path_to_CLI470>\bin\dev_appserver.py 
--runtime_python_path=<path_to_python311>\python.exe 
--python_virtualenv_path=<path_to_my_app>\myvenv311\ 
--datastore_path=<path_to_my_datastore>\db.datastore
<path_to_my_app>\

7- Changed first line of the application's app.yaml file to:

runtime: python311

I think that's it. Now I have the applications in the same PC still running on Python39+CLI431 (which still requires Python27) and also Python311+CLI470 sharing the same datastore and that's causing a minor issue that I'm documenting (asking for a solution) here.

Also, tested deleting (actually renaming) the env.var. CLOUDSDK_DEVAPPSERVER_PYTHON and Python39+CLI431 stops working, but running with Python311+CLI470 still works, so I guess that proves that in fact Python27 is no longer needed in dev.

[ludovich]

Gunicorn, the default web server used now in Python3 for AppEngine is not working on Windows:

https://github.com/benoitc/gunicorn/issues/524

so it would not start locally on such machine. This is the only reason it is not supported on Windows.

If you pick a different web server and your own entrypoint, it may work.

[micdearmas]

Thank you @ludovich for your response. I get the feeling you could be right because I think the patch from @NoCommandLine uses waitress as the webserver. However, I would need more than that because when I run dev_appserver.py, I don't specify a webserver and I'm not sure if that's possible.

Also, I've seen other conversations where people mention an entrypoint in their app.yaml, but that's something I've never used.

Could you please clarify that? Maybe provide an example of the batch command to run in order to start dev_appserver.py that way and how an entry point would look like in the app.yaml file (if it is really required and it is there where it goes) for a "regular" Flask app because the one for my apps look mostly like this one.

Thanks.

[NoCommandLine]

Yes, the patch replaces 'gunicorn' with 'waitress' because as @ludovich says, 'gunicorn' doesn't work on Windows.

The patch also fixed a few other stuff (the name of the folder for the executable, creating a copy of your requirements.txt file) but Google has adopted some of these fixes in later releases of dev_appserver.py.

If your app.yaml file doesn't specify an entrypoint, dev_appserver.py defaults it to 'gunicorn' (see this & this)

  .....NoCommandLine ......

 https://nocommandline.com

       Analytics & GUI for 
App Engine & Datastore Emulator

[ludovich]

See public entrypoint doc in https://cloud.google.com/appengine/docs/standard/python3/runtime#:~:text=If%20you%20do%20not%20speci....

When there is no entrypoint, the GCP Buildpacks layer generates a default one depending on the runtime:

See Buildpakcs references to gunicorn

https://github.com/search?q=repo%3AGoogleCloudPlatform%2Fbuildpacks%20gunicorn&type=code

Default is generated here https://github.com/GoogleCloudPlatform/buildpacks/blob/d43f8f8295840af201f9c68df893ae561f91969d/cmd/...

[micdearmas]

Thank you for those links, but I should say that I think the platform should maintain a higher level of abstraction (like it always did...until it didn't), ensuring that the internal specifics are managed behind the scenes. Users like me shouldn't have to deal with the underlying complexities (that I would have to deal with now), as these details were previously abstracted and should continue to be in order to focus on development, not infrastructure.

Around half of web developers use Windows as their base environment and while that might be a lot less than the share they had in 2008 when App Engine was released (and clearly targeted that OS), I don't know why GCloud is making it difficult now. They certainly have the resources to make it easier for us.

[ludovich]

I am more a Java guy, and AppEngine Java runtime was able to hide the complexity wth moving from jetty6, 9, and now 12.

Java is also very portable across versions and machines (no native code), and very advanced core libraries.

Python2 upgrade to Python3 is quite complicated per se, then the Python2  AppEngine Web Server was an internal one (webapp2) which was not ported to Python3. The Python team decided to default to gunicorn, good choice, except it does not work well on Windows. (Yet: I am following the gunicorn bug, they are making progress, you should add yourself and tell them you really want Windows support).

The fact that a popular Python Web Server does not work yet on Windows is still mind blowing to me.

Not a very enterprise ready ecosystem imo:-)