GCP automatically create GKE VM instances, project not uses GKE; GCP charge payments for it

[pmichalak]

Hi! I would like to ask a question, in best case answered (and solved) by GCP support. I am a supervisor in high school and I recommended students to use GCP for their projects. One group of students had a project in GCP; easily: 1 VM for API server (micro instance), 1 bucket for files, 1 MySQL database. Students were asked to not touch anything else (and they did not). However once we get a warning email from GCP that our instance is possibly used for high resources draining actions (??!). When we look at VMs in Compute engine, there was a new instance named as "gke-prod-cluster-....". We immediately removed this one and asked support for re-activate our project from stopped state and to answer "where is this instance came from??". They just activated a project again and did not answer any question. Students had another duties and they stopped also their API VM (to not drain money). BUT starting March one student (the one who own billing account) received an invoice for more than 100 EUR. We check VMs in Compute engine and there were, again, 2 "gke-prod-..." instances running AND AUTOMATICALLY CREATED in 22 of February! They were running and nobody asked for them, and they were draining money from STUDENTS! Now we immediately stopped whole project in GCP and trying to solve this mystery. Students (<18 years old) definitely not have over 100 EUR for unexpected issue from GCP side. They expected to pay up to 5 EUR for storage and static IPs, they even always start and stop their API VM when not needed. We even never activated GKE service! Please help me to solve this mystery, I feel really guilty for recommending GCP to students for their project, and they definitely never use GCP again, after this experience.

I am also including screenshots proofs:

 

[pmichalak]

I also want to mention, that there were no HA enabled for their VM and database (single region, smallest possible instances). Also they have no paid support (obviously), so this forum is the only way how to share this issue and possibly fix it. Not sure what to do now - pay it? Do not pay it? School can't pay this (unexpected) expenses for students.

[DamianS]

Hi,

Did you maybe checked logs under the project where GKE was created? 
SImple query will show activities . Search for ClusterManager.CreateCluster entry :

resource.type="gke_cluster"

log_name="projects/YOUR_PROJECT/logs/cloudaudit.googleapis.com%2Factivity"

results:

Regarding API. API can be disabled/enabled, so recommend to check it also under Logs Explorer (search by container.googleapis.com). If API was disabled ALL the time, it is not possible to create GKE cluster and MIGs related to him. 

cheers,
DamianS

[pmichalak]

Hi Damian! Thank You for your reply.

In order to not mess anything else and not drain more money, we unlinked billing from project and started 30 days deletion process. Thus seems logs are no more accessible. However I bet guys from GCP hold all those logs anyway. 

Actually GKE was not enabled or (re-disabled) over time, so it must be technical issue in GCP side. I made bunch of different very similar projects in GCP over past years and had no problem like this one before. I wonder if that might be some internal GCP mess up (which resulted to asking money from students).

Thanks!

Patrik