Internal server error when doing a datastore import

[timo-clipchamp]

Hi there,

We are backing up a production database to GCS daily. In a separate project, in order to verify that the backups are working, we are importing the backup from GCS to datastore. However, I am getting an internal error when doing so: 

ERROR: (gcloud.datastore.import) Internal error

The total size of the backup is pretty large (over 16TB), so I'm wondering if there's some limit to the import?

[robertcarlos]

Hi @timo-clipchamp,

Welcome to Google Cloud Community!

Would it be possible to provide a minimal, reproducible sample code that we could test and/or steps on how we could reproduce the issue?

Hoping to hear from you soon.

[timo-clipchamp]

Hi there,

Thanks for your reply. As it's mostly something set up in GCP, it's a bit difficult to provide a proper reproduction.

You need to set up 2 projects, one with a datastore with lots of data that exports to a GCS bucket. Then on the other project trigger the import for this bucket: 

gcloud datastore import gs://{path-to-exported-datastore-backup}.overall_export_metadata

Cheers

[robertcarlos · Accepted solution]

Hi @timo-clipchamp,

Please check if the proper roles are assigned to your account. You may check this documentation on Exporting and Importing Entities.

If everything checks out based on the documentation, you may want to wait for a few minutes before redeploying the command. You can also file a bug so that our engineers could take a look at this. We don't have a specific ETA for this one but you can keep track of its progress once the ticket has been created.

Hope this helps.