Page cleaner took x ms started to appear in logs

[PedroIt]

Hi,
We have this database in mysql server v8.0.18-google for some old eccommerce software for quite some time now.
All of the sudden, it started to show in logs these messages "Page cleaner took x ms to flush x and evict 0 pages". We did no software upgrade or settings change. And these messages appear about every 10-12 hours.
As a result, our webserver becomes overloaded with waiting connections (another thing i have to change) and then i have to restart it.
What i'm wondering is why there were no messages before 2024-07-31 ? And all of the sudden is a recurring thing. I even restart mysql server. But still the same.
Thank you

[ms4446]

The recurring "Page cleaner took x ms to flush x and evict 0 pages" messages in your MySQL logs suggest that the InnoDB storage engine is having trouble keeping up with writing modified data to disk.

Several factors could be causing this. An unexpected increase in database activity might be putting more strain on InnoDB, making it harder to manage and write data. Alternatively, there could be a drop in disk I/O performance, which means your system is slower at writing data to disk, causing these delays. The buffer pool, where data is temporarily stored before being written to disk, might also be too small to handle the workload, leading to more frequent and slower flushing.

Over time, your database tables and indexes might have become fragmented, which can make disk operations less efficient and slower. Even though you didn’t make any intentional changes, there might have been automatic adjustments in your cloud environment, such as resource allocation or background processes, that are affecting performance.

To address this issue, start by checking your disk I/O performance to see if there are any slowdowns. Increasing the InnoDB buffer pool size, if possible, might help by reducing the need for frequent data flushing. Additionally, reviewing and optimizing any long-running queries could reduce the workload on InnoDB. It’s also a good idea to monitor any background processes, like backups or analytics, that might be running at the same time you see these log messages.

Since this problem began suddenly, it’s important to investigate whether there were any changes in your cloud environment or underlying infrastructure around that time. By exploring these possibilities, you can identify the root cause and find a solution to improve your database’s performance.

[PedroIt]

Thank you for your reply.

There were no major changes to our usage.
In the last 30 days, the graph is constant:
 - CPU usage below 50% except when these errors occur, that spike to 80%-100%.
 - Disk read/write below 0.2 k/s when limit is 5.85k/s
 - Total memory usage is near 100%. trying to have everything in memory.

I'm out of ideas. What do you suggest i'd do? I was thinking trying to upgrade to the next version of mysql. And if that doesn't work, move to another instance.

Is there anyway to contact google to check if this is a cloud/instance performance issue?

[ms4446]

Given the stable usage patterns and the fact that CPU spikes only occur alongside page cleaner errors, it's unlikely that your MySQL issue is due to simple resource allocation within the current instance. To address this, there are several steps you should consider.

1. Contacting Google Cloud Support should be your priority. By sharing specific error messages, resource usage graphs, and details of your troubleshooting efforts, Google Cloud Support can leverage their deeper diagnostic tools to identify any underlying infrastructure issues that might be affecting your instance.

2. Upgrading MySQL is a logical step, but it should be approached with caution. Before upgrading, ensure you have a full backup of your database. It's also advisable to test the upgrade in a separate environment, created from a snapshot of your current instance, to validate compatibility with your e-commerce software and observe if the page cleaner issue persists. Consider performing an incremental upgrade rather than jumping to the latest version immediately to minimize risk and isolate potential problems. If the issue continues after the upgrade, inform Google Cloud Support, as this could provide valuable information for diagnosing any specific interactions between your software, the MySQL version, and the underlying infrastructure.

3. Moving to another instance should be considered only as a last resort. This step is complex and should only be taken if all other avenues have been exhausted. Careful planning and thorough testing are essential to avoid disruptions during the migration.

Additionally, you should analyze page cleaner details to identify any specific tables or data patterns that could be contributing to the issue. Reviewing your application logs for errors or slowdowns coinciding with the MySQL issues can also provide insights, as a specific operation might be triggering the page cleaner behavior.

[PedroIt]

How do i contact Google Cloud Support? We don't have paid support, and to be honest, it shouldn't be needed for something like this.