Help with gcloud artifacts docker upgrade migrate: Repository Not Found and No Images Copied

[DorelLine]

Hi Google Cloud Community,

I'm having trouble migrating images from Container Registry (GCR) to Artifact Registry using the gcloud artifacts docker upgrade migrate command. Here's what I tried and the issues I'm facing:

When I list repositories in my Container Registry using:

gcloud container images list --repository=eu.gcr.io/PROJECT_ID

I see the following output:
eu.gcr.io/PROJECT_ID/[FOLDER_NAME1]
eu.gcr.io/PROJECT_ID/[FOLDER_NAME2]
...
Each folder/container contains all my docker images

However, when I attempt to migrate using:

gcloud artifacts docker upgrade migrate --copy-only --projects=PROJECT_ID

The command outputs:

PROJECT_ID: Successfully copied 0 additional tags and 0 additional manifests. There were 0 failures.
Basically it copied nothing.

I tried the command with different parameters:

--from-gcr=eu.gcr.io/PROJECT_ID
--to-pkg-dev=PROJECT_ID/REPOSITORY_NAME

However, the migration fails with the error.....Repository "REPOSITORY_NAME" not found.
This happens even though I manually created the repository via the Google Cloud Console.

What i would like to achieve is copy all the images from the container registry to the artifact registry and keep the folder/repo structure like in the container registry.

I also created a simple bash script which:

1. runs gcloud container images list --repository=eu.gcr.io/PROJECT_ID
2. for each [FOLDER_NAME] creates a new repository at the artifact registry

But how do i migrate the docker images "automatically" by running the migration command provided by gcloud? Am i using the wrong parameters?

[mokit]

Hi, @DorelLine.

Please follow the instructions below for your scenario.

To create a repository for each folder in your GCR:

for folder in $(gcloud container images list --repository=eu.gcr.io/PROJECT_ID --format='value(name)'); do
  repo_name=$(echo $folder | sed 's/.*\///')
  gcloud artifacts repositories create $repo_name --repository-format=docker --location=EU --project=PROJECT_ID
done

Migrating Docker Images to Artifact Registry:

gcloud artifacts docker upgrade migrate \
  --from-gcr=eu.gcr.io/PROJECT_ID \
  --to-pkg-dev=projects/PROJECT_ID/locations/eu/repositories/REPOSITORY_NAME \
  --copy-only

Breakdown of the parameters:
--from-gcr: The source GCR repository.
--to-pkg-dev: The destination Artifact Registry repository, with the format projects/PROJECT_ID/locations/REGION/repositories/REPOSITORY_NAME.
--copy-only: Copies the images without upgrading them.

Or Automating the Migration for Multiple Folders:

#!/bin/bash

PROJECT_ID="PROJECT_ID"
REGION="EU"

for folder in $(gcloud container images list --repository=eu.gcr.io/$PROJECT_ID --format='value(name)'); do
  repo_name=$(echo $folder | sed 's/.*\///')

  gcloud artifacts repositories create $repo_name --repository-format=docker --location=$REGION --project=$PROJECT_ID

  gcloud artifacts docker upgrade migrate \
    --from-gcr=eu.gcr.io/$PROJECT_ID/$repo_name \
    --to-pkg-dev=projects/$PROJECT_ID/locations/$REGION/repositories/$repo_name \
    --copy-only
done

Regards,
Mokit

[DorelLine]

Hello @mokit 

Thanks for replying, unfortunately the parameter values are still invalid.

I have a very similar script:

#!/bin/bash

# Set the GCP project ID
PROJECT_ID="PROJECT_ID"

# List all repositories in the GCR
REPOS=$(gcloud container images list --repository=eu.gcr.io/$PROJECT_ID --format="value(NAME)")

# Loop through each repository
for REPO in $REPOS; do
  # Extract the last part of the repository name
  REPO_NAME=$(basename $REPO)

  # Create a new Artifact Registry repository
  echo "Creating Artifact Registry repository for $REPO_NAME in europe-west3..."
  
  read -p "Press enter to continue."
  
  gcloud artifacts repositories create $REPO_NAME \
      --repository-format=docker \
      --location=europe-west3 \
	  --mode=standard-repository \
      --description="Migrated from $REPO" \
	  
  gcloud artifacts docker upgrade migrate \
    --from-gcr=$PROJECT_ID/$REPO_NAME \
    --to-pkg-dev=projects/$PROJECT_ID/locations/europe-west3/repositories/$REPO_NAME \
    --copy-only

  if [ $? -eq 0 ]; then
    echo
    #echo "Successfully created repository: $REPO_NAME"
  else
    echo "Failed to create repository: $REPO_NAME"
  fi
done

And it always fails at the command

gcloud artifacts docker upgrade migrate

Right now the failing reason is:
--from-gcr must be of the form {host}/{project}
If i change to --from-gcr=eu.gcr.io/$PROJECT_ID , i get the failing reason:
--to-pkg-dev must be of the form {project}/{repo}
Setting all together:

  gcloud artifacts docker upgrade migrate \
    --from-gcr=eu.gcr.io/$PROJECT_ID \
    --to-pkg-dev=$PROJECT_ID/$REPO_NAME \
    --copy-only

has the output:
Copying images to europe-docker.pkg.dev/PROJECT_ID/REPOSITORY_NAME...
ERROR: gcloud crashed (TokenRefreshException): Bad status during token exchange: 404
b'{"errors":[{"code":"NAME_UNKNOWN","message":"Repository \\"REPOSITORY_NAME\\" not found"}]}\n'

I think now the issue relates to "europe-docker.pkg.dev"
If you check my script, the repo got created using location "europe-west3"
In order to specify the target location i could use parameter:
--pkg-dev-location=europe-west3

But the script, including the new parameters,  will "duplicate" all the images across multiple artifact repositories - because --from-gcr is pointing to the entire project, correct?
That would be not the desired outcome

FYI: Im using Google Cloud SDK 507.0.0

[DorelLine]

Can be closed.

Solution:
Simply create 1 artifact repository and copy all the images from the previous container registry:

gcloud artifacts docker upgrade migrate \
--from-gcr=eu.gcr.io/$PROJECT_ID \
--to-pkg-dev=$PROJECT_ID/$REPO_ID \
--copy-only

It will no split the folders by artifact repositories but thats fine, i will use a workaround.

[mokit]

That's wonderful to hear that @DorelLine 🎉

[jkolb]

The main problem here is that you keep adding an extra --copy-only flag in all these commands.

All you need is gcloud artifacts docker upgrade migrate --projects=PROJECT_ID which will do the whole migration for you in place including preserving urls.

The --copy-only flag is generally only used as one part of a more complex manual migration (not usually recommended).