Re: How to run dev_appserver with 'ascii' as the d...

micdearmas

I recently got my dev.env. to work with the latest version of dev_appserver (CLI 470) on Python 3.11 and things are looking pretty good so far. However, I keep getting a warning that says this when I start dev_appserver.py:

The default encoding of your local Python interpreter is set to 'utf-8' while App Engine's production environment uses 'ascii'; as a result your code may behave differently when deployed.

There are two reasons why I would like to address this warning:

1- The obvious one: Testing things in dev.env. should mimic prod.env. as much as possible.

2- I'm using a local datastore created previously with Python39 and some queries do not return anything. I think queries that use built-in indexes are ok, but if the query requires a composite index where one of the properties is a String property it just doesn't return anything. If I were to re-load those entities to the datastore using the newer Python version (3.11) then it's all good. I could be wrong, but I think this behavior could be related to that warning.

The good news is that in production all works well, but to avoid surprises I would like to run things in dev. as they are in prod. That is, running dev_appserver with 'ascii' as the default encoding.

I did some searching and I've tried this (I work on Windows):

set PYTHONUTF8=0
<path_to_python311>\python.exe -Xutf8 <path_to_CLI470>\dev_appserver.py ...

but I keep getting the warning and said behavior for some queries against datastore created with Python39 (the latter I'm not completely sure is related)

I also, ran this:

<path_to_python311>\python.exe <path_to_CLI470>\dev_appserver.py -h

but I don't see any flag or option to set the encoding.

Does anyone know if there's a way to run dev_appserver with the encoding as in production (ascii) for this version of CLI and Python or newer?

Thanks.

micdearmas

Just an update:

A new version (472.0.0) was released yesterday where they did some cleanup after removing Python2, but I just tried it and I'm still seeing this issue.

From the lack of responses I’ll assume that while App Engine for Python defaults to 'ascii' encoding (according to that warning from dev_appserver.py) in production environments, in dev. env. (using latest versions of Python and the SDK) is not possible to run dev_appserver with 'ascii' encoding.

I'll still go ahead and carefully migrate my apps to latest SDK/CLI + Python 3.11, and keep an eye out for new releases of App Engine for Python.

NoCommandLine

1) The error you see can be found in devappserver2.py and that check/error is also present in the gcloud SDK versions that needed Python2 to run dev_appserver.py

2) The default encoding in Python2 was ascii and so Google using ascii in production made sense (at least to me). I believe that check was there to warn you if you happened to have set your encoding to something else.

3) In Python2, you could set (change) the default encoding by running the following code

import sys
sys.setdefaultencoding("utf-8")

It turns out that lots of people were changing the default encoding to utf-8

4) Python3 was released with a default encoding of utf-8 and sys.setdefaultencoding() was removed in Python3

5) I've seen documentation that talks about making changes to your Python env by creating a sitecustomize.py file. I tried that but still got the warning (the one you got)

6) There's also documentation about using PYTHONIOENCODING environment variable. I tried that but still got the warning.

7) This documentation mentions that ASCII is valid utf-8. It says

A string of ASCII text is also valid UTF-8 text.

I don't have a 'definitive' answer to your question but maybe you can look at seeing if it's possible to encode to ascii before storing to datastore. Also check if the issue you face is due to the pickle protocol version

......NoCommandLine ......
https://nocommandline.com

Analytics & GUI for
App Engine & Datastore Emulator

micdearmas

Thank you @NoCommandLine , I think I'll keep the course because:

1- I already migrated some apps in dev. and deployed to prod. and no problems that I could see...so far.

2- Apparently, this is not as simple as setting a switch or environment variable as I thought it'd be.

It's just a matter of being careful for now and always be aware of that difference between dev. and prod.

I think it'd be ok if it was the other way around because, as you implied, ASCII could be seen as a subset of UTF-8; then, if it worked in dev. it should also work (for most cases) in prod. However, this is not the case and I wonder how/when they are going to eventually move from ASCII to UTF-8 in prod. as that seems to be the future for all new Python releases.

I do hope it'll be a transparent thing for users like me 🙂

How to run dev_appserver with 'ascii' as the default encoding in newer versions of Python?