Cannot SSH due to full disk

[csi_infra]

My ssh connection suddenly failed and I found something like this in the log: 

Jul 5 04:05:12 my-machine google_guest_agent[1157]: Creating user max
Jul 5 04:05:12 my-machine google_guest_agent[1157]: ERROR non_windows_accounts.go:134 Error creating user: useradd: /etc/passwd.2988: No space left on device
Jul 5 04:05:12 csi-nginx google_guest_agent[1157]: useradd: cannot lock /etc/passwd; try again later..

I did enhance the size of boot disk but it didn't work. How can I clear up some space if I can't get in to the shell at all.

[DamianS]

Hi,

You could try create new temporary machine in the same region as your "broken one". Then attach boot disk from "broken one" as additional disk to temporary machine and boot up VM. You should be able to mount this disk as normal mount and perform some cleanup. This approach will also let you fix e.g errors with fstab entries or ssh. If you need detailed info, just let me know. 
PS: Both VMs must be off, when you attaching disks. 

cheers,
DamianS

[DamianS]

Hi,

or you can also use serial port. Should work even if your VM is out of space. Grab the procedure 🙂
https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console#gcloud_2

cheers,
DamianS

[VishalBulbule]

In such cases it is suggested to use serial ports. To enable serial port acccess first enable it and then follow belos steps

• Go to the Google Cloud Console.
• Navigate to the VM instances page.
• Locate your instance and click on the instance name.
• Under the "Connect" tab, click on "Connect to serial console".
• This will open the Serial Console, where you can interact with the instance's console.
• Once you have access to the Serial Console or shell, identify and delete any unnecessary files or directories that are consuming disk space

https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console?_ga=2.457...

[baccione]

How do I log in if I do not know the credentials to log-in?

[8connor]

Did you figure out the solution for this?

[baccione]

Umh, not really. A colleague had a terminal open on the VM, so was able to solve the problem, and only afterwards I was able to set the password just in case for next time.

[DamianS]

Hello @8connor  ,Welcome on Google Cloud Community.
Please try this: https://medium.com/google-cloud/cloud-chronicles-google-cloud-in-action-fix-your-vm-in-a-minute-5580...

--
cheers,
Damian Sztankowski
LinkedIn medium.com Cloudskillsboost Sessionize Youtube