Migrate local datastore.db data created by python 2.7 dev_appserver.py to Datastore emulator

[KWagdy]

Hello, we have an old set of data locally created for our tests on dev_appserver.py python 2.7 application.

We are moving to Python 3, part of which it uses the new datastore emulator, which is using a new database file local_db.bin with a new format.

How can we migrate our local datastore.db file data to the Datastore emulator format?

We tried making a copy of the datastore.db file, and starting the emulator to the containing directory of the .db file, but it did not pick it up

$ gcloud beta emulators datastore --data-dir=<path to datastore.db>

[robertcarlos]

Hi @KWagdy,

Welcome to Google Cloud Community!

You can migrate your local datastore file to Datastore emulator by checking out this documentation on migrating to the Cloud Datastore Emulator.

Please be advised that this feature is still in Pre-GA Offering Terms, meaning that it could be identified as “Early Access,” “Alpha,” “Beta,” “Preview,” or “Experimental” and still in its development phase.

If you encounter any bugs or needed any additional features to be available in the future, you may file either a bug or a feature request through this issue tracker link. 

[KWagdy]

Hello, 

Thank you for your reply, I went through that page, the dev_appserver.py is able to read the datastore.db file correctly when using the --datastore_path=<PATH to datastore.db>

However, I was trying to use the Datastore Emulator gcloud beta emulators datastore --data-dir=<path to datastore.db> with the new Python3 recommended development approach by running python main.py rather than dev_appserver.py as some parts of the documentation recommends.

I eventually moved back to using dev_appserver.py rather than python main.py, but it would be good if the Datastore emulator can pick up the old datastore.db format instead of starting from scratch or having to migrate from Cloud to local

[robertcarlos]

Hi @KWagdy,

I totally understand that. As I've mentioned earlier, this feature is still its in Pre-GA Offering Terms, meaning that it could be identified as “Early Access,” “Alpha,” “Beta,” “Preview,” or “Experimental” and some of its features is still in its development phase. You may also check this Datastore Emulator reference which contains all the flags that are available for this function and their description. Also, here are the known issues of Datastore Emulator so far:

• The emulator simulates eventual consistency, by default. Firestore in Datastore mode is strongly consistent.
• The emulator does not allow non-ancestor queries within transactions. Firestore in Datastore mode no longer has this limitation.
• The emulator does not support IN, !=, and NOT-IN queries.
• The emulator does not support aggregation queries like COUNT(*).