Frequent NVIDIA drivers reinstall needed on boot

[ManuelMeterian]

Hi. I have a couple of Compute Engine instances that make use of a GPU and were provisioned with the Deep Learning ML images. They work fine most of the time.

But sometimes, after restart, the NVIDIA drivers won't load. And I must manually reinstall them following the usual instructions. Now, reinstalling them always fixes the problem. But the challenge is that I cannot automate the boot the start-stop of the machine, because every time I boot it up is a lottery. I never know when is it gonna fail.

• Did any of you experience a similar issue?
• Is there away to make this problem go away?

Thanks.

[JuatonCJ]

Hi @ManuelMeterian,

Welcome to Google Cloud Community!

Here are some guide to troubleshoot NVIDIA drivers connected to your issues:

• Automate the VM installation. You can utilize the scripts to fully automate the installation process. By implementing the scripts, you can reduce manual intervention across multiple installations or reboots.
• Make sure that you always use the latest image version, as older versions may have security vulnerabilities or other critical bugs.
• Choose a Deep Learning VM image family based on the framework and processor that you need. You may be asked to upgrade to a framework version that is within the supported time period.
• Monitoring VM lifecycle events can track VM events (like shutdowns and reboots) by creating a Cloud Monitoring dashboard. The dashboard helps you see system events and admin activities.
• Make sure that the latest NVIDIA driver is installed and running.

You may also check this document for best practices in building reliable systems on Compute Engine. It offers general tips and explains features that can help reduce downtime and handle unexpected VM failures.

I hope the above information is helpful.