Artifacts Registry Yum remote repositories (CentOS/Rocky)

[cesan3]

I have an environment where we can't access upstream yum/dnf repositories for centos and rocky. 

We've setup the remote repositories in our artifacts registry.  Creation succeeds but the repositories are empty.  Only the yum repodata files are downloaded and can be shown via `gcloud artifacts files list --repository <repo>`.  We've been waiting to see if the packages are populated but nothing happens.  We don't have policies or restrictions that limit connectivity.  

I'd like to know if there's something that triggers the mirroring process?  Can we look at some logs regarding that process?

If I add the artifacts registry repos in my Compute VMs,  yum/dnf fail building the cache.  The only file that is downloadable from the VM if proper Bearer token is used is the 'repodata/repomd.xml', the gzipped files, even if `gcloud` command show them, are not downloadable.

Am I missing something?

Thanks

[cesan3]

To complement and give more context:  If we check the files list for that repo, we get:

gcloud artifacts files list --repository rocky-8-baseos --location us
FILE CREATE_TIME UPDATE_TIME SIZE (MB) OWNER
repodata/05a0bdc4e3c1e51c54da18a3b8dfde27bbdefd35c0a835bfdf9f75a99a278390-filelists.sqlite.xz 2023-12-06T09:15:59 2023-12-06T09:15:59 1.557
repodata/250cccd2157e74be44cd52ffc2d77a50fafeadf59445ba4f62fe18f3ff2e810d-primary.sqlite.xz 2023-12-06T09:15:58 2023-12-06T09:15:58 2.120
repodata/6037eb97b3f2072d9edc3804b2254a620cd1551e12d3ec9944a7bec16e92624b-other.xml.gz 2023-12-06T09:15:57 2023-12-06T09:15:57 0.831
repodata/675c5355f1a4099ae1287e4b73bb5142bd01671448656baeebab5aded90f031d-primary.xml.gz 2023-12-06T09:15:56 2023-12-06T09:15:56 1.725
repodata/71f62d6dadfbf3238ce701da43cb69958ce4c546cc370f92e70ba933f3193c23-comps-BaseOS.x86_64.xml 2023-12-06T09:16:00 2023-12-06T09:16:00 0.285
repodata/76c4eebcc49b1cdf7fc494eb89473dca8eaa386a418f3c172af7b94e205550c8-other.sqlite.xz 2023-12-06T09:16:00 2023-12-06T09:16:00 0.387
repodata/d31187eb588eeba7b8a6a2d837a3acf0b9de85a1b6735885df2c29859def86ff-updateinfo.xml.gz 2023-12-06T09:16:01 2023-12-06T09:16:01 0.201
repodata/e105891d2832b712e68b45a603e895845e4df1c99d988936f02d3e899f68b5e5-comps-BaseOS.x86_64.xml.xz 2023-12-06T09:16:01 2023-12-06T09:16:01 0.054
repodata/f3ccdcd8d08444c2d446a2941f688a1827adcf50335019c689a2cd443645e2be-filelists.xml.gz 2023-12-06T09:15:57 2023-12-06T09:15:57 1.644
repodata/repomd.xml 

If we try to get the files providing authenticated token, the only file we can get is repomd.xml:

gcurl https://<repo-base-path-in-artifacts-registry>/rocky-8-appstream/repodata/repomd.xml > repomd.xml

cat repomd.xml
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
<revision>8.9</revision>
<tags>
<distro cpeid="cpe:/o:rocky:rocky:8">Rocky Linux 8</distro>
</tags>
<data type="primary">
<checksum type="sha256">a5eb798844e409b181e1c5b4876bb989295c0589e80c4b5aa29f3b17600b7e80</checksum>
<open-checksum type="sha256">01ab5cfbb636ba601ce20858ec1413f92eb7a2421cc35c33a1321b3f5f938c7b</open-checksum>
<location href="repodata/a5eb798844e409b181e1c5b4876bb989295c0589e80c4b5aa29f3b17600b7e80-primary.xml.gz"/>
<timestamp>1701887449</timestamp>
<size>1963127</size>
<open-size>19786083</open-size>
</data>
<data type="filelists">
<checksum type="sha256">341046ccf292ab60fe7be06d10c522d93a4163889f05508c34334828a96a5aaf</checksum>
<open-checksum type="sha256">b4bc62626f4db1fac9a6afa1211609479121306ede3b42c056c77495dbfe1a8b</open-checksum>
<location href="repodata/341046ccf292ab60fe7be06d10c522d93a4163889f05508c34334828a96a5aaf-filelists.xml.gz"/>
<timestamp>1701887449</timestamp>
<size>6621230</size>
<open-size>89588826</open-size>
</data>
<data type="other">
<checksum type="sha256">6fcbe488f9f5d86edcd3d08bd92b394d154411d53517ccfb38edd6389b26cb5d</checksum>
<open-checksum type="sha256">b5f7f4a3dfaabfae31c63b91c3964b89ae2a25b1cd9af04673d89d6e73319358</open-checksum>
<location href="repodata/6fcbe488f9f5d86edcd3d08bd92b394d154411d53517ccfb38edd6389b26cb5d-other.xml.gz"/>
<timestamp>1701887449</timestamp>
<size>1243961</size>
<open-size>14568952</open-size>
</data>
<data type="primary_db">
<checksum type="sha256">09d2ceeaacc3d106a320c0abe8caa160d253b6d83236e229c77ca274ce0edf96</checksum>
<open-checksum type="sha256">51d844691a12def15722b8850d29e0c73410e614ace68d94c674469e4fd5bb02</open-checksum>
<location href="repodata/09d2ceeaacc3d106a320c0abe8caa160d253b6d83236e229c77ca274ce0edf96-primary.sqlite.xz"/>
<timestamp>1701887458</timestamp>
<size>3262868</size>
<open-size>20709376</open-size>
<database_version>10</database_version>
</data>
<data type="filelists_db">
<checksum type="sha256">6f588b7b4715fd646ad154b0c8cb7eb10bac24420e88be43c73bbadcd8d6ed48</checksum>
<open-checksum type="sha256">bca5092616e2341cc3349c3de9acec9ca2ca9548b4045153747bb9db95ec8648</open-checksum>
<location href="repodata/6f588b7b4715fd646ad154b0c8cb7eb10bac24420e88be43c73bbadcd8d6ed48-filelists.sqlite.xz"/>
<timestamp>1701887463</timestamp>
<size>5204076</size>
<open-size>40615936</open-size>
<database_version>10</database_version>
</data>
<data type="other_db">
<checksum type="sha256">a2528c75f34491ed945de4895fc7d6b468426dbb1cacf18755ec072a2afcf8fd</checksum>
<open-checksum type="sha256">ac55a5f8884e9152965ff2dbd80dd4ceae097e1daaca989f9e3119569a32bd2a</open-checksum>
<location href="repodata/a2528c75f34491ed945de4895fc7d6b468426dbb1cacf18755ec072a2afcf8fd-other.sqlite.xz"/>
<timestamp>1701887453</timestamp>
<size>1329180</size>
<open-size>12574720</open-size>
<database_version>10</database_version>
</data>
<data type="group">
<checksum type="sha256">ec2fa1982439f4b0ca7dc86f882f18657457a2d42f06ac6718298c7f7528df43</checksum>
<location href="repodata/ec2fa1982439f4b0ca7dc86f882f18657457a2d42f06ac6718298c7f7528df43-comps-AppStream.x86_64.xml"/>
<timestamp>1701887443</timestamp>
<size>486316</size>
</data>
<data type="group_xz">
<checksum type="sha256">7ba43f88671d8107d6603ebff822fc071e73596a1e20779855af577b0e6e343a</checksum>
<open-checksum type="sha256">ec2fa1982439f4b0ca7dc86f882f18657457a2d42f06ac6718298c7f7528df43</open-checksum>
<location href="repodata/7ba43f88671d8107d6603ebff822fc071e73596a1e20779855af577b0e6e343a-comps-AppStream.x86_64.xml.xz"/>
<timestamp>1701887449</timestamp>
<size>81840</size>
<open-size>486316</open-size>
</data>
<data type="modules">
<checksum type="sha256">b6d3922872dea41ea2621cbd71d792c097ab02105a52d23527cc9fe32f761423</checksum>
<open-checksum type="sha256">d15e8b7197adf4e5ddf96a8feb1c49835407cb75c7c0d8a8551105e0a15927eb</open-checksum>
<location href="repodata/b6d3922872dea41ea2621cbd71d792c097ab02105a52d23527cc9fe32f761423-modules.yaml.xz"/>
<timestamp>1701887537</timestamp>
<size>74188</size>
<open-size>687459</open-size>
</data>
<data type="updateinfo">
<checksum type="sha256">6b018c498f01763000ec08cfaeb27d550a887480750466cde20ee303f67f57bf</checksum>
<open-checksum type="sha256">972f8d7db0be306a31fbcd1364cc8b2eaf2d4e166a36141e80b8249d18251f77</open-checksum>
<location href="repodata/6b018c498f01763000ec08cfaeb27d550a887480750466cde20ee303f67f57bf-updateinfo.xml.gz"/>
<timestamp>1701888191</timestamp>
<size>824553</size>
<open-size>6435028</open-size>
</data>
</repomd>

the others report that the entity is not found:

gcurl https://<repo-base-path-in-artifacts-registry>/rocky-8-appstream/repodata/d0dd9f0998bdbdc2a5bcaac1f29eacbfe1e3de63db131b827fc48041b78a9c6f-filelists.xml.gz > test.xml.gz

cat test.xml.gz
Requested entity was not found.

Any thoughts?

[tony-ferrum]

We have a similar situation, we manually mirror the yum repos ourselves. Have found GAR to be less than reliable for this task
1. boot a VM
2. `reposync`
3. `createrepo`
4. Upload all repos to a single GCS bucket
5. Run a small Go binary on the systems that need access, configuring yum to point at this localhost:port/ as the baseurl

[cesan3 · Accepted solution]

So, finally, the problem was found with a recent documentation update.

The problem is the baseurl that the tool suggests using.

gcloud beta artifacts print-settings yum --repository rocky-8-appstream --location us
# To configure your package manager with this repository, do the following:

# Prepare you VM to access the repository using the following instructions:
# https://cloud.google.com/artifact-registry/docs/os-packages/rpm/configure#prepare-yum

# Configure your VM to access Artifact Registry packages using the following
# command:

sudo tee -a /etc/yum.repos.d/artifact-registry.repo << EOF
[rocky-8-appstream]
name=rocky-8-appstream
baseurl=https://us-yum.pkg.dev/projects/PROJECT-ID/rocky-8-appstream
enabled=1
repo_gpgcheck=0
gpgcheck=0
EOF

# Update Yum:
sudo yum makecache

# For complete setup information, see
# https://cloud.google.com/artifact-registry/docs/os-packages/rpm/configure

Checking current documentation references to use '/remote' path instead of '/projects'

sudo tee -a /etc/yum.repos.d/artifact-registry.repo << EOF
[YUM_REPOSITORY_NAME]
name=REPOSITORY_NAME
baseurl=https://LOCATION-yum.pkg.dev/remote/PROJECT_ID/REMOTE_REPOSITORY_NAME
enabled=1
gpgcheck=1
EOF

[wmcg]

Cant believe this problem has not yet been fixed. Finding this solution marks the end of a painful debugging process - thank you!