europe-west9-a still not working

Hielko · 05-12-2023 02:26 AM

According to Google Support (https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY) the problems with the Paris datacenter (europe-west9-a) are solved. They mention that users with remaining issues are contacted by Google. Sadly this is not the case for us. We run a Compute Engine on west9-a, and the server is running, we can stop and start it. CPU is running continuously on 36,73%. We can't access the instance with ssh, while we can access our other CE's in other regions. Creating a snapshot fails also (timeout).

We don't have a paid Google Cloud support contract, and we think it shouldn't be needed for this case. The datacenter outage clearly has some remaining issues, which should be solved by Google for all customers. We contacted Google Cloud support by Twitter, but they don't offer any help.

Any idea what we can do now?

Willbin

Hello @Hielko,

Welcome to Google Cloud Community!

It would be best to be in touch with the Cloud Platform Support so they can further inspect your specific project level issues.

You may want to choose the right services for your organization on the link below.
https://cloud.google.com/contact

Thanks!

Hielko

Thanks @Willbin , the problem is that we first have to start a paid support contract before Google is willing to help with a problem situation caused by the datacenter outage in Paris. Google acknowledges that there are remaining issues, but only wants to help if you have a paid support contract. That's not ok in my opinion, I don't ask support for just a technical issue, but for restoring the basic functionality that is missing due to the datacenter outage. I'm lost and I think we need to move to another cloud provider that does acknowledge responsibility for technical issues that are completely external to the customer.

Willbin

Hello @Hielko,

I understand your situation.

Can you send me the error that you are encountering when accessing your instances with ssh? I will see what I can do on my end to help you or reproduce this. Thanks

Hielko

Thanks for your kind reply @Willbin I appreciate it. The compute engine can be stopped and started without problems. But when we try to connect with ssh (just like we always did with this instance before the datacenter outage, and we still do with other compute engine instances), we get the following error screen:

Our permissions are ok, port 22 is allowed on all our instances. When we do the Troubleshoot, all checks are ok:

When we run gcloud compute ssh <our-server-name> --zone europe-west9-a --troubleshoot

all results are ok.

When we run gcloud compute ssh <our-server-name> --zone=europe-west9-a --tunnel-through-iap

the connection is denied:

When we try to create a snapshot for the (bootable) source disk of this instance, the snapshot creation fails without specific error.

Is it just a coincidence that there is an ongoing issue in the west9 region: https://status.cloud.google.com/incidents/zCfpbUaJA8hk5EdD69wK#73mBtVKKfeJGJ1yaY7hV

If there is no easy solution and it just our server in the west9-a zone that has still problems then we will stop trying to get access to the server and delete the instance. In that case we will lose some data, that would be really sad. The last weeks we checked every update on the west9-a outage, and we were so happy that the cloud status page showed a green status finally.

Hielko

Hi @Willbin

Just to show the response of the snapshot request:

Creating snapshots for our other instances (not in west9-a) works normally.

Edutal

We have the same for our compute instances in europe-west9-a, and agree that Google Cloud Support is terrible compared to other providers like AWS. At least if such a drama happens (we are already more than 3 weeks unable to access our instances), you would expect emergency support, paybacks and maybe even free credits as a compensation.

Hielko

Thanks @Edutal for joining this thread. I just posted a link to this thread on Twitter https://twitter.com/appyhapps_nl/status/1658398481365467136?s=20 I loved the Google Cloud stability and reliability, but with this issue I discovered that it is not possible to report any technical issue to Google Cloud support, that's ridiculous.

Hielko

Hi @DamianS We pay for our Google Cloud compute engines. I even pay now for a compute instance in west9-a that I can't reach. There is no way to contact Google Cloud support to report this issue, they just refer to the publick support pages and this forum. The issuetracker is not for Google Cloud customer support.

I don't have a problem paying for support that I need for issues that I have. But if there is a technical problem caused by Google or under responsibility of Google (like the datacenter outage), and there is no way to report an issue related to this, then something is clearly lacking. With Azure in my previous job, we were able to report always a technical issue. The response time was not great, but it was possible to tell Microsoft that they had a problem.

DamianS

Yep, but as I said, when you've decided to use GC, you agreed way of support they providing. I know that sucks. I totally agree with you, however it is as it is. According to that, complaining that something is wrong is not cool for me. Apart from that, did you've tried to set up new instance in this region, rip off disk from unaccessible machine and attach as additional disk into newly created ?

Hielko

When the data outage started, Google advised customers to start a replacing instance. But this wasn't possible during the first hours after the data center went down. We had no way to report this to Google Cloud support. Now Google says that everything is working again, and that they contacted customers with remaining issues. But they didn't contact us. This is why I am complaining. This is a lack of support because Google doesn't take responsibility for problems and solutions they offer.
I will try your suggestion, it is a boot-disk but hopefully it will work. I will post it here if it solves the access problem.

By the way, I understand your comment about backup but we can't backup every second. There is data on the disk that has changed since the last backup that is worth recovering.

DamianS

Most probably they contacted only with ppl who have paid support, which is normal I would say. Maybe it's stupid, however, did you've tried to login via ssh from gcloud instead of using UI?

Hielko

gcloud and ssh same issue, not possible to connect

DamianS

Did you've checked the logs for this VM? I'm not talking about screenshots from UI, but live logs, when you've trying to login via UI button SSH.

Hielko

I created another compute instance with the existing boot-disk of the instance we can't connect with, and it was again not possible to connect with this new instance. When we create another fresh instance in the same zone, we could connect with it without problems. We we added the existing disk to that new instance as a second disk, we couldn't retrieve a uuid of this disk, we couldn't mount it.

I think the disk is corrupt in some way. We will stop trying, in the end it costs more time and the only possible solution is (I think) when a technical engineer from Google Cloud support looks into it.

Hielko

@DamianS thanks for your suggestions.