How to solve "No space left on device" error without root per ssh

[e_ruider]

Hello,

I am already quite desperate so I am happy about any suggestions to keep my data.

I have a Compute Engine VM running on SUSE Linux Enterprise Server 15 SP1 which is used to host a  SAP CAL instance. The VM was created using the SAP CAL UI, so I can only have root access via a private SSH key and I do not have a root password. The main problem is that I copied too many files onto the boot disk so it ran full. I enlarged the disk via GCloud Console but unfortunately, it did not resize the sda3 partition so I still have the "No space left on device" error. The instance now does not boot properly so I can only access it via the serial console where I can only login as a specific non-root user. Therefore, I can't use any sudo commands.

Do you have any hint on how I can either delete the files (not owned by the only user I can login with) or regain root access (startup scripts don't work for some reason) or anything to make the instance boot properly again? Once more, I am happy about any suggestions as I have tried and failed with the usual recommendations.

[DamianS · Accepted solution]

Hello @e_ruider  ,Welcome on Google Cloud Community.

You can do following things:

Workaround 1:
* Create VM with the same OS in the same zone.
* Stop SAP VM.
* Detach disk.
* Mount detached disk to new VM
* Boot and mount temporary
* Remove files. 

Workaround 2:
Try this instruction https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console#set-up-pa... 

I've been fixing many VMs on AWS and GCP by workaround 1 🙂 
If you need more details, let me know. 

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost Sessionize Youtube

[e_ruider]

Hi @DamianS , thank you for your suggestion. As I had already tried (and failed with) Workaround 2, I'd like to try Workaround 1.

Unfortunately, the very same OS version is not available anymore (I run SUSE Linux Enterprise Server 15 SP1). Do you think it would work with 15 SP2? Or is it possible that I cannot re-mount the disk to my old instance then?

[DamianS]

@e_ruider It should work with SP2. The main point here is, that VM must be in the same region as your "broken" disk. 

[e_ruider]

@DamianSI have now created an instance in the same region which does boot fine without the full disk attached. However, when I try to start the instance with the disk attached as a secondary disk, it does not boot, throwing the "no space left on device" error. How can I solve this?

[e_ruider]

@DamianSall good, I have used the serial console to connect as root and to delete the files. Thanks a lot, this worked like a charm.

[DamianS]

@e_ruider Fantastic news 😄 Maybe I should create some video about troubleshooting GCE in a funny way 😄