Artifact Registry migration created +100GB of tmp entries that I'm now paying for.

[Toby1]

I used the Artifact Registry migration tool suggested by the App Engine team's emails 8 days ago. Today I did a deployment to validate that it was working correctly, and saw in Artifact Registry that there was +100GB of imagines in us.gcr.io, and when I looked at them they're all of the format " app-engine-tmp/app/[APP NAME]/ttl-[time range]" where time range is 2h, 18h, or 7d.

It appears they were made as part of the migration script provided by the App Engine team. I don't know if they're still necessary should/could/or can be deleted. In the mean time billing has kicked in, and I'm now paying for the tmp files created by the migration process. That's pretty frustrating.

Has anyone run into this issue?

[jkolb]

These files are part of app-engine and are not related to the migration script (These files were either already in GCR befor the migration tool was run or they were created by app-engine deployments after your migration)

[Toby1]

Are you sure? The us.gcr.io updated field says "8 days ago" which corresponds to the day I ran the migration script. The Artifact Registry Storage bill went from $0.006 / day to $0.336 / day on the day I ran the script (8 days ago). Also there were no deployments that day, so no other reason for anything to change in Artifact Registry.

[jkolb]

Yes, running the script copies them from Container Registry to Artifact Registry. They weren't previously in Artifact Registry, they were in Container Registry.

In the Artifact Registry UI, you should be able to see the "Created" date for these images, which will tell you when they were created.

[Toby1]

While they were created in different dates, all of the images in us.gcr.io were updated on the date when the script was run (now 9 days ago). Which coincides with the day Artifact Registry Storage billing increased 56x.

If not the script that was run, what App Engine process updated these images, and can they be deleted? Since usage of Artifact Registry is entirely managed by App Engine, is there a process to clean up unused images?

[Toby1]

Actually, all of the repositories under us.gcr.io were created on the same date as the same day script was run, while the images themselves were created earlier. So repositories (app-engine-tmp/app/ttl-2h, app-engine-tmp/app/ttl-2h/default/buildpack-app, etc.) were all created at the same time the script was run.

Are they still needed? Was there a ttl mechanism that was meant to delete them?

[Toby1 · Accepted solution]

I went in and hand deleted what I think are unused images, hoping that it doesn't cause any problems. I cannot tell, because there is no documentation on why or how the images are being used by App Engine.

It's frustrating that Google App Engine didn't clean up after itself. It's also frustrating that the migration instructions didn't make clear that this would increase cost, nor provide suggestions for mitigation. I'm left concerned that any future forced migrations will also be poorly handled and secretly increase cost.