Firestore emulator restore data from cloud Firestore

[JoshRTP]

This use to work, but it has stopped working.

The process I use is the same as this:

https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-loc...

Export from Firestore

Download export

Start emulator with: firebase emulators:start --import ./folder-name

I get this error in the logs:

Apr. 12, 2023 8:32:22 A.M. com.google.cloud.datastore.emulator.firestore.CloudFirestore main
SEVERE: Exiting due to unexpected exception.
com.google.cloud.datastore.core.exception.DatastoreException: Message missing required fields: kind_info[0].kind
	at com.google.cloud.datastore.util.leveldb.ExportImportUtil.parseBackupFile(ExportImportUtil.java:365)
	at com.google.cloud.datastore.util.leveldb.ExportImportUtil.fetchEntities(ExportImportUtil.java:85)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.init(CloudFirestore.java:177)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.startLocally(CloudFirestore.java:111)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:92)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: kind_info[0].kind
	at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
	at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:73)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:96)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at com.google.cloud.datastore.util.leveldb.ExportImportUtil.parseBackupFile(ExportImportUtil.java:363)
	... 4 more

All the terminal indicates is there is an unexpected error and is shutting down.

System: Pop!_OS 20.04 LST

Firebase: 11.26.0

[christianpaula]

Hi @JoshRTP,

Welcome to Google Cloud Community!

The error message suggests that there might be an issue with the exported data that you are trying to import into the Firestore emulator. To troubleshoot this issue, you can double-check that the export was successful and not corrupted, check compatibility between the version of the Firestore emulator and Firebase SDK, or use the gcloud command-line tool to import data into the Firestore emulator.

Thanks

[JoshRTP]

Thanks for your response. I've been looking into this for a couple week and you are the first to answer.

Since the Firestore emulator and SDK are the latest versions I'm not sure how to check find out if the versions are compatible. Is there a tool or list somewhere?

I can't find information on how to import directly from firestore into my emulator. 

I have used the CLI to export the backup into a bucket, then copy the backup to my local environment, but I still get the same problem. Is there a guide somewhere that shows how to import data from firebase directly into the emulator?

[ChromeQ]

I ran into this exact same issue. Unfortunately I couldn't easily fix it, but my work around was to add `--collection-ids='admin','days','exclusives','pins','users'` where these are my collection names. Try adding one collection at a time as a process of elimination if it fails for you.

I noticed from this the output directory contains 5 outputs (zero indexed) - when leaving off the collection-ids it exports _all_ collections and I notice 6 output named files. 

Seems as though there is an additional "hidden" collection in my firestore which is incompatible with the emulator. I have a theory it could be a firebase analytics collection but I may be wrong. But analytics have been enabled a long time and my script worked until quite recently, updating to latest versions didn't help.

Did you solve the issue @JoshRTP ? 

[nanea-it]

Same issue. Starting from 4th October we experienced this problem (I've manually checked all the backups and until 3nd october all went fine). It's not about size of DB because I've the same error in our mirror-test-project with few data.

I've noticed that without any apparent reason, the all_namespaces/all_kinds/output-N files are 10x more than previous backups. But the MB of exported data is not changed.

We added a new index the previous week, but the issue happened 5 or 6 days after this operation.

As @ChromeQ mentioned, if I set for the backup all the collections explicitly then the emulator works fine.

[DrMandelbrot]

Hello, I'd like to know if you solved this? I tried to downgrade firebase and firebase-tools but it didnt work.
I also tried to manually indicate the collections I want, but I have subcollections who are not included because they are generated dynamically.
Any news?

[JoshRTP]

I didn't get a solution, and I have switched projects so I am not spending more time on looking for one. If you want to use all the data in a large dataset you'll need to setup a firestore database project and point your dev environment at it. Then you can create backups and restore them in that project.

The emulator fails when the dataset gets too large. I don't know the exact size, but it wasn't dependent on the machine I was using.