Hi, I am currently using TPUs through TPU Research Cloud program.
A few days ago, my billing account was closed due to reaching the budget limit. Following this, the running TPUs should be automatically stopped when the account was closed.
However, my TPU quota usage hasn't been released. Furthermore, some IP addresses corresponding to the TPU VM instances are still in used. It seems that the TPU VM instances are not properly stopped and they are not captured correctly.
When I tried to see the details by clicking the VM instances appeared at the IP addresses, I received the below message:
----------------
Unable to find the resource you requested
There was an error while loading /compute/instancesDetail/zones/us-central2-b/instances/t1v-n-4a822891-w-1?project=sodium-ray-397604&hl=ko&inv=1&invt=AbkuAA. Please try again.
It may be a browser or network issue. Go to the loading issues help page to troubleshoot the issue.
Request ID: 1011601426561817819
----------------
I tried to remove the TPU VMs but I cannot see any instances by using gcloud CLI (Listed 0 items). So I disabled the TPU API for resetting the corresponding resources, but the VMs are not removed and now I cannot enable the TPU API again with the below message:
----------------
$ gcloud services enable tpu.googleapis.com --project sodium-ray-397604
ERROR: (gcloud.services.enable) The operation "operations/acf.p2-316433763239-e5bd1997-a0af-4d95-9b56-7fbde6adb554" resulted in a failure "[The enablement process failed for service 'tpu.googleapis.com'.
Help Token: Af9utURtE-fTzAdntB3tvazvZ3LbrHwBSVggl9MwDnmZoK6ozvNvsqoOggPbzviUsmBx_fi2uGaW4ABDojpBFwkyyFaRfGfaLTCSJ5n9XEO9RS3v] with failed services [tpu.googleapis.com]".
Details: "[<DetailsValueListEntry
additionalProperties: [<AdditionalProperty
key: '@type'
value: <JsonValue
string_value: 'type.googleapis.com/google.rpc.PreconditionFailure'>>, <AdditionalProperty
key: 'violations'
value: <JsonValue
array_value: <JsonArray
entries: [<JsonValue
object_value: <JsonObject
properties: [<Property
key: 'type'
value: <JsonValue
string_value: 'googleapis.com'>>, <Property
key: 'subject'
value: <JsonValue
string_value: '?error_code=160003&service=tpu.googleapis.com'>>]>>]>>>]>]".
----------------
Please help to enable my TPU API and remove the invisible zombie TPU VMs. These issues are causing delays in our research progress.
Thank you.
LinkedIn
Twitter
