The server freezes after an automatic 'Google Cloud SDK' update

[john45456]

The server freezes after an automatic 'Google Cloud SDK' update, making SSH and HTTP-web inaccessible. A stop/start of the server is the only thing that helps, and then everything works again. This situation has occurred twice in a row, and every time the server freezes, the log shows some dnf automatic update 'sequence'.

How can this be resolved? Should these updates be disabled?

Thank you for your advice

/var/log/messages:

Sep 24 13:13:43 server systemd[1]: Starting GCE Workload Certificate refresh...
Sep 24 13:13:43 server gce_workload_cert_refresh[297198]: 2024/09/24 13:13:43: Done
Sep 24 13:13:43 server systemd[1]: gce-workload-cert-refresh.service: Deactivated successfully.
Sep 24 13:13:43 server systemd[1]: Finished GCE Workload Certificate refresh.
Sep 24 13:13:50 server systemd[1]: Starting dnf makecache...
Sep 24 13:13:51 server dnf[297204]: Rocky Linux 9 - Cloud Kernel 20 kB/s | 3.4 kB 00:00
Sep 24 13:13:52 server dnf[297204]: Extra Packages for Enterprise Linux 9 - x86_64 52 kB/s | 41 kB 00:00
Sep 24 13:13:53 server dnf[297204]: Extra Packages for Enterprise Linux 9 openh264 3.0 kB/s | 993 B 00:00
Sep 24 13:13:53 server dnf[297204]: Google Compute Engine 18 kB/s | 1.4 kB 00:00
Sep 24 13:13:53 server dnf[297204]: Google Cloud SDK 15 kB/s | 1.4 kB 00:00
Sep 24 13:13:55 server dnf[297204]: Google Cloud SDK 68 MB/s | 131 MB 00:01
Sep 24 13:14:01 server systemd[1]: Created slice User Slice of UID 0.
Sep 24 13:14:01 server systemd[1]: Starting User Runtime Directory /run/user/0...
Sep 24 13:14:01 server systemd[1]: Finished User Runtime Directory /run/user/0.
Sep 24 13:14:01 server systemd[1]: Starting User Manager for UID 0...
Sep 24 13:14:02 server systemd[297216]: Queued start job for default target Main User Target.
Sep 24 13:14:02 server systemd[297216]: Created slice User Application Slice.
Sep 24 13:14:02 server systemd[297216]: Mark boot as successful after the user session has run 2 minutes was skipped because of an unmet condition check (ConditionUser=!@system).
Sep 24 13:14:02 server systemd[297216]: Started Daily Cleanup of User's Temporary Directories.
Sep 24 13:14:02 server systemd[297216]: Reached target Paths.
Sep 24 13:14:02 server systemd[297216]: Reached target Timers.
Sep 24 13:14:02 server systemd[297216]: Starting D-Bus User Message Bus Socket...
Sep 24 13:14:02 server systemd[297216]: PipeWire PulseAudio was skipped because of an unmet condition check (ConditionUser=!root).
Sep 24 13:14:02 server systemd[297216]: Listening on PipeWire Multimedia System Sockets.
Sep 24 13:14:02 server systemd[297216]: Starting Create User's Volatile Files and Directories...
Sep 24 13:14:02 server systemd[297216]: Finished Create User's Volatile Files and Directories.
Sep 24 13:14:02 server systemd[297216]: Listening on D-Bus User Message Bus Socket.
Sep 24 13:14:02 server systemd[297216]: Reached target Sockets.
Sep 24 13:14:02 server systemd[297216]: Reached target Basic System.
Sep 24 13:14:02 server systemd[297216]: Reached target Main User Target.
Sep 24 13:14:02 server systemd[297216]: Startup finished in 329ms.
Sep 24 13:14:02 server systemd[1]: Started User Manager for UID 0.
Sep 24 13:14:02 server systemd[1]: Started Session 13310 of User root.
Sep 24 13:14:02 server systemd[1]: session-13310.scope: Deactivated successfully.

[END OF LOG BEFORE SERVER FREEZES]

[reinc]

Hi @john45456, 

Welcome to Google Cloud Community!

If you're experiencing issues with updates (like the freezing you mentioned), disabling updates temporarily can help pinpoint whether they're the cause.

1. Try this command to disable automatic update checks and it allows you to maintain the current version of the Google Cloud SDK.

gcloud config set component_manager/disable_update_check true 

2. To re-enable automatic update checks use this command

gcloud config set component_manager/disable_update_check false 

3. If you already disable update and want to manually update your Google Cloud SDK version use this command:

gcloud components update --list 

Here are also the possible compatibility issue: 

• Missing Out on New Features: Updates often introduce new features and capabilities for the SDK and Google Cloud services. You might not be able to use these features if your SDK is outdated.

• Compatibility Problems: Newer Google Cloud services might not work properly with older SDK versions, leading to errors or unexpected behavior in your applications.

• Dependency Conflicts: The SDK relies on other software packages (dependencies). Updates might change these dependencies, and older versions might not work with newer Google Cloud services.

You can check here the Google Cloud SDK - Release Notes for a more updated version and Rocky Linux documentation about automatic updates

I hope the above information is helpful.

[john45456]

The same situation happened again so I'm sure it's caused by these automatic updates, I'm sending the end of log before server freezes

Sep 29 06:48:11 server systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Sep 29 06:48:11 server systemd[1]: Stopped User Runtime Directory /run/user/0.
Sep 29 06:48:11 server systemd[1]: Removed slice User Slice of UID 0.
Sep 29 06:48:29 server systemd[1]: Starting dnf makecache...
Sep 29 06:48:29 server dnf[317212]: Rocky Linux 9 - Cloud Kernel 22 kB/s | 3.4 kB 00:00
Sep 29 06:48:30 server dnf[317212]: Extra Packages for Enterprise Linux 9 - x86_64 55 kB/s | 36 kB 00:00
Sep 29 06:48:31 server dnf[317212]: Extra Packages for Enterprise Linux 9 - x86_64 25 MB/s | 23 MB 00:00

[END OF LOG / SERVER FREZE HERE ]

 The command

gcloud config set component_manager/disable_update_check true 

doesn't work, I ran it (and it successfully set to true) but today I still see this automatic update in the log

Sep 30 07:17:02 server systemd[1]: Started User Manager for UID 0.
Sep 30 07:17:02 server systemd[1]: Started Session 3906 of User root.
Sep 30 07:17:02 server dnf[53659]: Extra Packages for Enterprise Linux 9 - x86_64 31 kB/s | 15 kB 00:00
Sep 30 07:17:02 server systemd[1]: session-3906.scope: Deactivated successfully.
Sep 30 07:17:02 server dnf[53659]: Extra Packages for Enterprise Linux 9 openh264 2.6 kB/s | 993 B 00:00
Sep 30 07:17:03 server dnf[53659]: Google Compute Engine 17 kB/s | 1.4 kB 00:00
Sep 30 07:17:03 server dnf[53659]: Google Cloud SDK 10 kB/s | 1.4 kB 00:00
Sep 30 07:17:05 server dnf[53659]: MariaDB 68 kB/s | 3.4 kB 00:00
Sep 30 07:17:05 server dnf[53659]: Rocky Linux 9 - BaseOS 17 kB/s | 4.1 kB 00:00
Sep 30 07:17:05 server dnf[53659]: Rocky Linux 9 - AppStream 21 kB/s | 4.5 kB 00:00
Sep 30 07:17:05 server dnf[53659]: Rocky Linux 9 - Extras 14 kB/s | 2.9 kB 00:00
Sep 30 07:17:06 server dnf[53659]: Metadata cache created.

The server freezes from time to time, it works for x-days without a problem, suddenly it freezes and before freezing it in the log is visible this automatic update, it's still the same scenario

Before I switched to Rocky linux, the server was running under a Centos linux distribution (in Google cloud), I never experienced a similar problem with "old" centos OS

How to completely disable these automatic updates?

Server OS- Rocky Linux 9.4

[reinc]

Hi @john45456 

To disable automatic updates on Rocky Linux 9.4, you need to use the OS-level tools. Here's how:
Disable the dnf-automatic Service:

sudo systemctl disable --now dnf-automatic.timer
sudo systemctl mask dnf-automatic.timer

To Verify:

sudo systemctl list-timers | grep dnf-automatic

The output should be empty, indicating that the timer is disabled.



I hope the above information is helpful.