BQ query speed in recent weeks?

[nickf]

Hello - so I've been using BQ to query a few-hundred-GB dataset for a few years now. Data is constantly inserted (and still seems to be with no issues), and with the exception of complex JOINs, queries run rapidly (like <30s, consistently).

I've been away for 2 weeks, and I come back to find even simple queries that would complete in 30s or less are running for 30minutes or more. Nothing has changed with the queries, data, tables etc.

Did I miss a fundamental change to how BQ works in the past few weeks, or is this perhaps a temporary issue that should pass?

[caryna]

Hi @nickf,

Welcome to Google Cloud Community!

I understand how frustrating it can be when your BigQuery queries are taking longer than usual to complete. Below are some suggestions that may help improve performance and resolve the issue:

• Performance Insights: It provides insights and shows you which parts of the query take longer to process.
• Resource Utilization: Ensure that your project’s resources have not changed, as this can slow down your queries. You can also use BigQuery Monitoring to keep an eye on how much you're using. 
• Optimize Query Computation: You can apply various optimization techniques to improve your query performance. For detailed information on best practices and optimization methods, please refer to the relevant documentation.
• Use history-based Optimizations: This uses information from already completed execution of the same query to optimize and improve the subsequent query. You may refer to this documentation on how to apply history-based optimizations.
• Data Skew: Verify if there have been any changes in the distribution of your data. Uneven data distribution can cause certain nodes to become overloaded, which may lead to slower query processing. Ensuring a more balanced distribution can improve query efficiency.

If the issue persists, I suggest contacting Google Cloud Support as they can provide more insights to see if the behavior you've encountered is a known issue or specific to your project.

For future reference, you can check the Google Cloud Service Health Status and release notes to see if there are any known BigQuery service outages in your area and check the latest updates or new features that might affect how well your query runs.

I hope the above information is helpful.