Migrating App Engine apps from Python27 to Python 3 runtime (a datastore question)

[micdearmas]

I just learned that in a little less than a year I won't be able to make any changes to my (web) apps running in the Python27 runtime and that puts me in very difficult spot because I have not one, but several ones.
Because of this, I'm considering following the path of least resistance which I think it's to use legacy bundled services from the Python3 apps. Also, all of my apps in that situation are using the webapp2 framework and I've read that they recommend to first migrate to Flask and then do the rest of the migration.

Since some of the web applications I have to migrate are not small, and I don't know how I could have some requests (pages) being handled by webapp2 and others by Flask in the same app (so that I can do the migration step-by-step); I thought of creating (for each of my apps/project) a brand new project to work in the Python 3 runtime using Flask and progressively start migrating the handlers of each (page) request to this framework and also use legacy bundled services, so that I don't have to make too many modifications for this stage of the migration.

However, say one of the Google App Engine projects I have now running an app in Python27 is called AP27; I don't know if I could have new Google App Engine project say AP3 connecting and with full access to the datastore that is used by project AP27. Is that possible at all?

If it is, how do I go about that in the production environment (GCloud) and in the dev server? Right now each of my web apps (projects) have its own instance of Datastore in production and also locally in my home computer where I have the devserver. By the way, I don't use NDB just db.Model, so my code looks something like this:

from google.appengine.ext import db
#Define structure for table Comments		  
class Comments(db.Model):
  author = db.UserProperty()
  username = db.StringProperty()
  content = db.TextProperty()
  page = db.StringProperty()
  date = db.DateTimeProperty(auto_now_add=True)
  active = db.BooleanProperty()

Note: All projects would be under the same Google account. Also, I'm assuming that at any point I could tell App Engine that requests to mydomain.com be handled by project AP3 instead of AP27.

Any help or alternative approach suggestions are appreciated.
Thanks.

[NoCommandLine · Accepted solution]

1. Don't create a new project
2. Create a new version or service for your existing project. All Services/Versions of a project have access to the same datastore (see documentation).

For example, you could create a new service under your existing project to handle your Python 3 code (say you called this 'python3'). You can also distinguish it from your default (existing service/app) by using a url of the format /python3/<your_standard_url>/. Then you create a dispatch.yaml file to route all requests with that url pattern to your python3 service. (see documentation for creating dispatch.yaml files)

This way you have both the Python 3 and Python 2 code available for the same project and they have access to the same project resources (datastore, service accounts, etc). When you're done testing the Python 3 and are ready to switch over to it, you can make it the default service for your GAE Project 

Note: I suggest switching over to NDB. You can still use the bundled APIs (see documentation on enabling them for Python 3). Using Bundled APIs means you write less code. It also has the added advantage that while you're testing on your dev environment (using dev_appserver.py), you'll probably not be using production resources which might count against your free quota (i.e. some of the bundled APIs use a mock when you're on dev environment which means they're not connecting to production and so do not make production API calls; Cloud APIs make calls to production APIs which count against your quota)

Cheat Sheet for migrating from db to NDB

 ..... NoCommandLine ......

 https://nocommandline.com

A GUI for Google App Engine

[micdearmas]

Thank you @NoCommandLine , it'll take me a bit of time to go over all that info and resources you pointed out and I might follow up creating other questions as I go along.

There's one thing that caught my attention right way, though. You say I'll have to switch over to NDB, are you sure about that? I know that not all the modules/features available in Python27 will be available when using "bundled services" in Python3, but I think I saw somewhere (would have to find it again) that db.Model would be one of those supported when using "legacy bundled services". Isn't that the case?
Thanks again.

[NoCommandLine]

You're right, 'db' is still available in the legacy bundled APIs - see (here). I'll update my response from 'you'll have to switch' to 'I suggest switching' (Google 'shipped' and encouraged migration to NDB a long time ago before all the issue of migrating to Python 3 came up; since you're migrating, I was thinking it made more sense to also migrate to NDB) 

[cameron19]

We were in a very similar situation - very large application on Python 2.7, DB instead of NDB - and recently finished our migration to Python 3, making use of the bundled services (and staying on DB, for now).  I definitely agree with @NoCommandLine and recommend creating a new service in the same App Engine project for a Python 3 version of your application. 

I actually just wrote up a blog post documenting our entire journey, with our strategy, how we set up our dev environment, and all the things we learned along the way: 

How to Migrate from Google App Engine Python 2 to Python 3 with Minimal Risk

It was a daunting project, so I wanted to share what I figured out with anyone who still has a migration ahead of them.

Happy to answer more questions if they come up during your migration(s)!

-Cameron