Can't connect to VM due to full disk, after resizing, same error occurs in Google Cloud CLI

[hidrostix]

So i've had this problem for 3 days at this point. 
To start with, I wanted to run a Minecraft server on google cloud, and I used a mod to preload 3.5million chunks which should've taken about a few days in real life. I assumed that my 10GB boot disk got full and for some reason my VM didn't use the other 50GB disk. When I made a snapshot of both to trouble shoot, the disk was empty and the bootdisk only had 6.7GB / 10GB. 

Because of this weird error, I couldn't connect to SSH after fixing my firewall connections, serial console  would not start login service because of a full boot disk. 
So i downloaded google cloud CLI and tried connecting from my terminal, didn't work. Then it mentioned I had a full disk and it needed resizing. Which i have done multiple times while troubleshooting. 
I resized from 10GB to 50, then to 200, now at 202GB, and it still required more space. Is there some way to fix this error?
the google cloud CLIthe spammed message on my serial console

I had also tried resizing manually
making my boot disk into a snapshot, and then switching it into a disk and into the new bootdisk of my vm
recreated the VM and repeated the step above, still to no avail. 

[DamianS]

Hello @hidrostix  ,Welcome on Google Cloud Community.

Do you have only one disk and storing data on that one disk ? If you want to debug, you can follow this approach:

1. Create similar VM, at the same region and zone as you broken one.
2. Shut down your broken one, once done, detach boot disk ( or disk which is broken ).
3. Attach broken disk to new VM and boot it. Disk should be attached as additional one, not boot one. 
4. Mount your broken disk as normal disk by mount /DISK_PATH/ /mountPOINT
5. As root, you should be able to check what is going on with this disk, make some cleanup or resize it. 
6. Once done, shut down New VM. Detach disk , attach to old VM. Start Old VM.

You can use also serial console to debug this state:

https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console 

If you have any question, 

---

@hidrostix wrote:

I resized from 10GB to 50, then to 200, now at 202GB, and it still required more space. Is there some way to fix this error?

---

One question here: Did you've resized it from GC UI AND then from VM terminal, or only resized via GC UI ? 

--
cheers,
Damian Sztankowski
LinkedIn medium.com Cloudskillsboost Sessionize Youtube

[hidrostix]

So I only resized it from the GC UI, I am unable to access the VM terminal since I can't access SSH and serial console. This is because the bootdisk hasn't been resized in both GC UI and the terminal. But I need the terminal to resize it. So it's like a loop I can't break. 
The only solution I've thought of is to resize it through the startup script, but I don't know how to do that. 
And accessing or resizing with ssh through GC CLI is also not possible.
GC CLI error
I've tried using --troubleshoot --tunnel-through-iap. It gave the same error, saying that the VM needed more disk space.

It gave 2 links to resizing and trouble shooting with serial console. Both required ssh and serial console which I can't access.

I have also made sure my firewalls worked with checking the firewall and the SSH and TCP configurations in IAP (I don't remember the name but I think that was it)

But my ssh still gave the same error:

[DamianS · Accepted solution]

@hidrostix ,

Yep. SSH will not going to work due to full disk. What I've mentioned is using serial console, which is slightly different method of connecting with VM. So as I've mentioned in my previous post, you can either do following steps:

"1. Create similar VM, at the same region and zone as you broken one.
2. Shut down your broken one, once done, detach boot disk ( or disk which is broken ).
3. Attach broken disk to new VM and boot it. Disk should be attached as additional one, not boot one. 
4. Mount your broken disk as normal disk by mount /DISK_PATH/ /mountPOINT
5. As root, you should be able to check what is going on with this disk, make some cleanup or resize it. 
6. Once done, shut down New VM. Detach disk , attach to old VM. Start Old VM."

OR 

Reconfigure your VM to enable serial-console

https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console 

Please see this tutorial:

I’ve intentionally filled out disk space.

That caused following error:

So I’ve stopped VM. Expanded BOOT disk from 10 GB to 50GB And started VM again. Still the same

In theory I should be able to resize it from terminal in my case. But I want to simulate your case, so imagine that I’m not able to use terminal at broken VM.

1. I’ve stopped broken one.

2. Once done, I’ve detach boot disk.

3. then I’ve picked clean VM and attached EXISTING DISK ( boot disk from broken one)
   
4. Next, I’ve logged to clean VM.
   
5. fdisk -l presented that broken disk is being visible from OS perspective
   
6. As root, I’ve created mountpoint named “broken” and mounted device with type Linux filesystem
   
   
7. As you can see, we have all files from broken VM ( now you can clean FS or do anything).
   
8. Lets’ resize disk.

We will going to use two commands

growparth DEVICE PARTITION_NUMBER
resize2fs /FS

9. Umount /broken mountpoint. Shutdown VM

10. Detach disk
    
11. Configure new disk on broken VM

12. save config and start VM.

Boot disk has been fixed

--
cheers,
Damian Sztankowski
LinkedIn medium.com Cloudskillsboost Sessionize Youtube