Cloud SQL Instance stuck on "Instance being updated" status

[vladz25]

Our development Cloud SQL Instance stuck on "Instance being updated" status for more than 12 hours.  The insight shows that we have reached our SLA limit ang recommended that we  split the database across multiple instances or turning off the innodb_file_per_table flag. I tried restoring the backup to a new instance but that failed after waiting for more than 3 hours. I was not able to change the innodb_file_per_table flag since all operations are disabled when an instance is in this state.

I also noticed that the daily backup was not working on "daily" basis.

[ms4446]

It sounds like your Cloud SQL instance is stuck in a state where it's unable to complete the update process. This can occur for various reasons, including reaching resource limits or having a large database with the innodb_file_per_table flag enabled.

Here are some steps you can try to resolve the issue:

1. Restart the Instance: Sometimes, a simple restart can clear up temporary issues preventing the update process from completing.
2. Contact Google Cloud Support: If the problem persists, it's a good idea to open a support ticket. A SQL specialist from Google Cloud can investigate and assist in resolving the issue.
3. innodb_file_per_table Flag: If you have this flag enabled, consider turning it off. This will store all InnoDB tables in the system tablespace, potentially improving performance and reducing the risk of update failures.
4. Split the Database: If your database is large, consider splitting it across multiple instances to distribute the workload.

Regarding the daily backup:

If the daily backup isn't running as expected:

1. Ensure the backup schedule is correctly configured.
2. Check the Cloud SQL logs for any errors or warnings related to the backup process.
3. If issues persist, contact Google Cloud support for further assistance.

Additional tips:

• Consider enabling the Maintenance window feature in Cloud SQL to ensure updates are applied during a predetermined time window.
• Regularly review your database size and workload to ensure your instance is appropriately sized.
• If using the innodb_file_per_table flag, monitor database performance to ensure it meets your needs.

[vladz25]

@ms4446 thanks for your suggestions.

• Restart the Instance & innodb_file_per_table Flag - not possible since all operations are disabled.
• Split the Database  - already tried by restoring the backup first (failed).
• Contact Google Cloud Support - this is the last option.

About the daily backup I think i know what causes it. Last few weeks I made a scheduler where all development instances will be shutdown when no one is using them and then be turned on  in the morning. The timing coincide with the daily backup - my bad 😁.

[ms4446]

I understand your situation. Given that you've identified the scheduler as the potential cause of the daily backup issue, here's what I recommend:

1. Adjust the Scheduler: Consider temporarily disabling the scheduler that shuts down your development instances. This should allow the daily backups to run without interruption. Once you've confirmed that the backups are running successfully, you can re-enable the scheduler. However, make sure to adjust its timing to avoid any conflicts with the backup process in the future.

2. Google Cloud Support: If the issue with the "Instance being updated" status persists or if you encounter any other challenges, I recommend reaching out to Google Cloud support. They have the tools and expertise to delve deeper into the problem and provide specific solutions.

[vladz25]

Update: Server finished the update after almost 24hours. I already applied the flag and added a new server. Hope that I never encounter this problem again. Thanks @ms4446