Connecting to existing (created using Python27) datastore locally from Python3 and bundled services

[micdearmas]

I've setup a dev environment on Windows 10 to start migrating Standard App Engine Python27 to Python3 using legacy bundled services.
It is the same computer where I have all my apps working on App Engine Python27, each with its own already populated local Datastore (and I'm using only db for all things Datastore, that's something I plan on modernizing once I have all apps migrated to Python 3 and Flask) e.g.:

from google.appengine.ext import db

I started with the home page of a relatively simple app and changed it to use Flask instead of webapp2; that's working.

The issue I'm running into now is that the new service/version of the app using Python3 doesn't seem to 'see' the existing local Datastore. It is ok on Production (GCloud) where I tested it deploying as a new service and it's also ok if I start populating a new (different file) datastore locally, so I thought I just might be missing maybe a parameter or setting an environment variable when I start the server locally.
This is what I'm doing to start it:

C:\Python27\python.exe "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin\dev_appserver.py" --runtime_python_path=D:\path-to-python3\python.exe --port=8092 --admin_port=8008 --datastore_path=D:\path-to-this-app-datastore\this-app.datastore D:\path-to-this-app-new-service-in-python3\

The file D:\path-to-this-app-datastore\this-app.datastore contains the Datastore for this app which I can use with no problems from the Python27 version of the app (I don't run the two versions at the same time), but the Python3 version doesn't return any records (entities). A couple of times I got this error 'No api proxy found for service "datastore_v3"', but then I don't know why I don't get it anymore, it just doesn't return any records.

The most interesting part is that if I start the server providing a different non-existing datastore file like this:
D:\path-to-this-app-datastore\this-app-new.datastore
and I ran some code to populate the table (Entity) then the very same code I'm testing against the datastore created using Python27 works fine. The data is also persisted and I can stop the server and ran it again and the data is there, just like one would expect. I just can't get it to return records from the 'original' datastore and now I don't even get any errors/exceptions.

I'm not completely stuck, but some of the other applications I need to migrate will be a lot of pain if I have to populate again a local Datastore.

Any ideas?
Thanks.

The app.yaml file contains this:

runtime: python39
app_engine_apis: true
service: this-app-python3

and the requirements.txt is this:

Flask==2.1.0
appengine-python-standard>=1.0.0

[NoCommandLine]

1. I suggest you check Stackoverflow for possible solutions
2. You can also try using the datastore emulator to export data from your existing DB and then import to a new DB (see documentation for export/import from emulator). However, you should note that gcloud exports data in LevelDB format which isn't human-readable. This means that if you're able to successfully export your data, it'll appear as gibberish to you if you try to view the data directly until you import it again into your local datastore and then view it in your App or datastore viewer (the next version of our App should include the capability to export data to JSON or CSV; the idea is to make it easier for our users to easily move their data around between environments or even view it outside a datastore). 
3. Another option is to try using NDB and see if it solves your problem i.e. find the smallest data model you can convert to ndb, do the conversion and see if it solves your problem. 

[micdearmas]

Got it, thanks.

[NoCommandLine]

If you're still looking for a solution to this, our App now handles it and there's a post up on our blog about it.

 ..... NoCommandLine ......

 https://nocommandline.com

A GUI for Google App Engine

[micdearmas]

Thank you for pointing that out.

I've been making do by generating scripts that pre-populate the local/dev datastore from the Python 3 version of the app (a bit of a pain).
So far, I've migrated some apps, but have a few more to go, so I'll keep that other option/tool in mind.

Then, after I have everything running on Python 3 (and no concerns about the January 2024 deadline), I'd like to  start moving away from legacy bundled services and for that I will have to make changes again to the way I access the Datastore because I think db.model will no longer be supported by the newer APIs that access the Datastore.

Hopefully, it will be changes only to the way I write/read the data and not to the data/Datastore itself, but that's still a few weeks/months away, so I guess I'll cross that bridge when I get there 🙂

Thanks again.

[NoCommandLine]

> I'd like to  start moving away from legacy bundled services

Keep this in mind
For now, there's currently an 'issue' with testing locally without the bundled service for ndb or datastore. When you use the ndb or datastore client (meaning you're not using the bundled API), it will ALWAYS connect to your Production data even when you're working locally.

However, if you use the bundled API, it will use data from Cloud Emulator when you're on dev environment and use Production data when it's running on GAE servers.

According to Google's documentation, this isn't supposed to be the behavior. ndb or datastore client should use data in your cloud emulator when you turn it on but this isn't happening now. Maybe they'll fix it by the time you need to migrate

 ..... NoCommandLine ......

 https://nocommandline.com

A GUI for Google App Engine