Encountered an auto deletion of VM Instance, Storage Buckets along with my other GCP resources

[mrn59]

Show More

My account just encountered an auto deletion of VM Instances, Storage Buckets and other GCP Resources. I am trying to find support with investigation for the same! 

Additionally, any help retrieving the lost resources would be much appreciated. For added information, We did not have Snapshots created for our VM Instances, Nor did we have Bucket Lifecycle Activated for any of our Buckets.

Thanks!

[KirTitievsky]

That sounds terrible. Sorry.  Can you tell us as much as you can about what you have observed?  Any emails or messages in the UI? Any messages you get when you try to use your existing accounts with the project?  What do you see in the UI when you try to switch to the project?

[mrn59]

Hi KirTitievsky, Thanks for following back! We do not see any aberrations with regards to the Project or the UI. However, this took place about 8+ hours ago. Deletions happened without our notice. Interestingly, our systems were down and to our surprise when we arrived at the console to look out for the same, we watched deletions taking place at the very moments - They were happening right in front of our eyes 😞

1) Additionally, we did notice cloud Log Explorer logging these activities as they happened. We couldn't make a lot out of what and how it happened reviewing those thoroughly, however an expert may be able to take a look and help better investigate what had happened here really!

2) Secondly, we tried all means available to get help from support, but to no avail. The id associated falls under my university account (they being the organizational admin). Upon reaching out to the university IT support, they mention their inaccessibility to provide support since in reality they aren't honored with Organizational Administrator duties by Google!

I'm happy to work with the support team privately if needed to regain back the lost resources!  

Thanks again!

[KirTitievsky]

Can you please go to console.cloud.google.com/logs and try to find log entries for these deletion operations. They should have a "principal email."  Please don't share the full email in this public forum, but see if you recognize the email address. I would go to https://console.corp.google.com/iam-admin/iam and see what permissions this principal has. If this is a project you own, it may be wise to take away all permissions from this account for now to prevent further deletions.  Let me know what you find out and we can investigate further. 

If you university has not purchased a support plan with GCP, it may be tough to get someone on the line in time.  

[mrn59]

Hi KirTitievsky, the "principal email" associated there shows my email id that is linked with this account and this project as 'Owner'. But then I haven't done any deletions or haven't done anything that would trigger the same. Not sure why this would still show this id!?

As far as support for the GCP is concerned, I'm happy to contribute my fair share separately to access support with regards to this. Another way in which I'm trying to avail the support is by creating a support request by creating a new separate account. Not sure if that would take me any far! 

Happy to take any quick resolution route you might suggest. The deletions has a major collateral damage with critical severity!

[KirTitievsky]

I would guess that your account may have gotten hijacked (or you did something unintentional).   Maybe create a new google account or use a colleague's as a new admin account and take your own permissions away as soon as possible. 

[KirTitievsky]

Once we stop the process, if it is possible to restore anything, you'll likely need a support contract and work through support to do that. Please start here: https://cloud.google.com/support

[KirTitievsky]

Note that you can find more information about who's doing this in the requestAttributes field of the Log entries.

[mrn59]

Yes we are trying to make most out of the Request sub-fields including requestAttributes

[mrn59]

For the same reasons stated above, I am unable to reach support (or create a support request due to missing Support IAM role) and hence I am here trying to make the most out of what I can.

Do you think you'll be able to get me in touch with the support directly? Also, Do you think creating a New GCP Account and trying to request Support from there would work - Esp. since they may or may not be able to access this account from there!?

[KirTitievsky]

Let's try that. I do not know of another way to put you in touch with support otherwise.  You might be able to raise this as an abuse issue .   

[mrn59]

The challenge is, even when I am the Project Owner, my account accesses seem quite restricted likely because of University (acting as Organization Admin) Policies. This readily makes me come across following restrictions trying to access most GCP resources:

"We are sorry, but you do not have access to <XYZ>. Please contact your Organization Administrator for access."

While the Project was originally created, Policies were quite liberal and allowed for flexible access to most of the GCP Resources. It's only since last 18-24 months (beginning some time in Oct 2021 - Mar 2022) that I've started coming across more such restrictions w.r.t. the GCP resource access. Because of this, Nor Am I able to add any New Google Account OR Use a Colleague's Account as a New Admin. I may be able to take away my own permissions, but that won't be ideal without any backups!

[KirTitievsky]

To make sure I understand you correctly: You are seeing all the resources being deleted using your identity.  But you do not have the administrator role on the project nor specific permissions to add support or remove users. The account is managed by "IT" who is refusing to help you, including granting you permissions to administer the project or buy a support plan?

If so, you might have another tool at your disposal: change your password at security.google.com. This, IIUC, should immediately invalidate any existing security tokens associated with your account and stop further deletions. 

[mrn59]

Alright - Password Changed! Abuse Reported! Tech Support Request Raised from a New Separate Independent GCP Account. Let's see how it goes - Fingers Crossed!!

[KirTitievsky]

Good. Sorry, I can imagine this was a bad day for you.  Let's hope this at least stabilizes things.