Cloud Run => Custom Domains => At most 15 SSL certificates

[nick-youngblut]

I have been using the "Custom domains" integration in Cloud Run to set up the load balancer with custom domains. This has worked well, up until I tried to deploy my 16th app to Cloud Run. I then received an error stating "At most 15 SSL certificates" (see attached). 

Why at most 15 SSL certificates? 

Why did I not receive any warnings when I approached the limit (e.g., at 13 SSL certificates, receive a warning when adding the 14th)? 

It appears that I cannot just use a managed certificate for a wildcard domain:

ERROR: (gcloud.compute.ssl-certificates.create) Could not fetch resource:
 - Invalid value for field 'resource.managed.domains[0]': '*.my_custom_domain.org'. Wildcard domains not supported.

...so do I now have to create managed ssl certificates for each individual app's custom domain? 

[nick-youngblut]

I created a new managed certificate for all domains via:

gcloud compute ssl-certificates create $CERT_NAME \
  --domains=$DOMAINS \
  --global 

I then updated the proxy via `gcloud compute target-https-proxies update`, and I deleted the old certifications via `gcloud compute ssl-certificates delete`. 

This worked once the new certification was provisioned (~20 minutes). 

However, when I try to create a new custom domain integration via the Cloud Run console (see the images in my first post), ALL of the ssl certificates for each individual deployed app are recreated. This results in the same `At most 15 SSL certificates` error. 

Do I have to stop using the Cloud Run console for creating custom domain integrations for newly deployed apps?

If so, there should be a big asterisk on the Custom Integrations section of the Cloud Run console that states "*really only works for <=15 apps".

Notably, there doesn't seem to be any way of editing the SSL certificates used for the custom integrations section of the Cloud Run Console (attached), so it is really unclear on what to do once one hits the 15 SSL certificate limit.

[AI]

Maybe not using the integration in this case could be beneficial. There are two types of certificates now in GCP - compute (aka classic, aka legacy type) and certificate manager certificates. The wildcard can be created only with the latter. What you could try is creating a wildcard certificate, then updating the proxy (at this moment certificate manager certificates can only be associated with your target proxy using gcloud command) and point the proxy at your cloud run using a serverless network endpoint group backend type.

[nick-youngblut]

Thanks AI for the quick response.

So, I creating integrations via the console really doesn't work beyond 15 Cloud Run services? 

It sounds like you are saying to remove all existing integrations (downtime for each app) and using your suggested approach. Can you point me to the relevant GCP docs on setting up integrations without using the Cloud Run GCP Console approach that I've been using (e.g., all of the gcloud commands required)? 

[AI]

---

@nick-youngblut wrote:

So, I creating integrations via the console really doesn't work beyond 15 Cloud Run services? 

---

In fact this has nothing to do with Cloud Run itself. The limit for 15 ssl certificates comes at the targetHttpsProxy level (currently, you may specify up to 15 Classic SSL Certificates) and is applicable to any load balancers using Compute/legacy certificates.

For creating a Certificate Manager certificate (which supports the wildcard) you can use this doc: https://cloud.google.com/certificate-manager/docs/deploy-google-managed-dns-auth. It also has a part about deploying the certificate to a load balancer.

The following doc explains creating Serverless Network Endpoint group based Load balancer: https://cloud.google.com/load-balancing/docs/https/setup-global-ext-https-serverless.

[AI]

The limit is also mentioned here along with the note about Certificate Manager certificates.

[nick-youngblut]

Thanks Al for all of the info. Do you know if the "Custom domains" console interface will be updated anytime soon so that developers don't run into this "dead end" in which they suddenly hit the SSL certificate limit (n=15) and then cannot change any of the relevant settings via "Custom domains" console interface? I'm surprised that the interface is not more flexible -- one can basically just map domains to a services, with no control over SSL certificates (or any other settings).

[nick-youngblut]

I'm trying to set up new domain mappings via gcloud, given that I've hit my limit via the "Custom domains" console interface. 

When I run:

gcloud beta run domain-mappings create \
  --service=my_new_service \
  --domain=my_new_service.my_domain.org

... I get:

ERROR: (gcloud.beta.run.domain-mappings.create) The provided domain does not appear to be verified for the current account so a domain mapping cannot be created. Visit [https://cloud.google.com/run/docs/mapping-custom-domains/] for more information.
You currently have no verified domains.

When I list the domain mappings via `gcloud beta run domain-mappings list --region=us-west1` (we use us-west1 for everything), I get `Listed 0 items.`. I get zero items whether I use my standard google account or my developer service account. 

I don't get why I can't see any existing mappings, given that I have 15 services mapped to domains. I've always used "Custom domains" console interface to set up the mappings, but shouldn't they be listed when I run `gcloud beta run domain-mappings list`?

Note: I can list all url-maps via `gcloud compute url-maps list` and target https proxies via `gcloud compute target-https-proxies list`. ...but an empty list is returned with `gcloud beta run domain-mappings list`.

[AI]

Have you tried verifying that domain (https://search.google.com/search-console/welcome)?

[nick-youngblut]

I've been using the parent domain ("my_subdoman.my_domain.org") for all 15 of my existing Cloud Run services (e.g., "my_service.my_subdoman.my_domain.org"). I've never had to validate the domain or subdomain before when using the "Custom domains" console interface. Why would I need to when using `gcloud` to map the service to the domain?

[AI]

Sadly I don't know the answer to this question, as I work with different products in GCP, but I suggest you open a public bug in GCP bug tracker: https://issuetracker.google.com/issues

[nick-youngblut]

I seem to be in the same situation as https://stackoverflow.com/questions/70510539/google-cloud-run-domain-mappings-from-cloud-build-servi...

From the post: 

The domain for which I'm setting the mapping is hosted also in Google Cloud (Cloud Domains) and dns are managed via Cloud DNS. As I said, using Web UI works like a charm.

Why does gcloud require the extra setup (e.g., "Added cloudbuild service account to the Google Search Console for given domain as an owner") when the web UI "works like a charm" without such extra setup (e.g., extra domain verification of ownership)?

[nick-youngblut]

I don't see any clear docs on how to perform the same steps as the Cloud Run Custom Domains mapping web UI (as noted above) but via the `gcloud` CLI. 

Such docs would be VERY helpful, given the 15 SSL cert limit for the web UI.

[nick-youngblut]

Now Google is completely getting rid of the Cloud Run integrations 🤣

I was hoping that I could use terraform to automate the complex CLI setup, but I would need to first obtain a terraform configuration for my existing Cloud Run setup. HOWEVER, terraformer does NOT support Cloud Run 🤣. I guess that it's time to find a better app hosting service.

If anyone has suggestions on alternatives, such as Heroku or Digital Ocean, I'd love to hear them.

[nick-youngblut]

So, I'm assuming that there are NO clearly described docs on replicating the Custom Domains GCP Console feature via the `gcloud` cli?

Developers are on their own?