Hi!
Once a day, I migrate data from the project to Bigquery.
But since May 8, right after loading the table, the InternalTableExpired method automatically deletes my new table. Settings set to "Table expiration - NEVER".
If your BigQuery table is being automatically deleted after loading the data, despite having the table expiration set to "NEVER," there could be a few possible reasons for this issue. Here are a few things you can check:
Check for Data Lifecycle Policies: Data Lifecycle Policies can be set at the dataset level or the project level in BigQuery. These policies can specify rules for data retention and automatic deletion. Verify if there are any data lifecycle policies set at either the dataset or project level that could be causing the table to be deleted.
Confirm Job Settings: When you load data into BigQuery, ensure that you review the job configuration settings. Double-check that there are no specific settings or flags (e.g., --expiration_time
) that may be causing the table to expire or be deleted after a certain period.
Review Custom Scripts or External Tools: If you are using any custom scripts or external tools to migrate data to BigQuery, make sure to examine those as well. It's possible that a script or tool might be explicitly deleting the table after the migration process. Review any cleanup or deletion steps in your scripts or tools that may affect the table.
Check with Other Users or Administrators: If multiple users have access to the BigQuery project or dataset, ensure that nobody else is inadvertently deleting the table or modifying its expiration settings. Collaborate with other users or administrators to investigate if any changes have been made recently that could affect the table.
Examine Audit Logs: BigQuery provides audit logs that track actions taken within the project. Check the audit logs to see if any actions related to table deletion or expiration have been recorded. This can help identify the source of the issue or provide additional context.
@ms4446 Thanks for the advice! I checked everything - nothing helps. The development team assures that nothing has been changed in the code.
Here's some more information, maybe it will help.
The problem started after May 8th.
I have many datasets but the problem is only with one that contains three tables.
Log name: cloudaudit.googleapis.com/system_even
Every day, all table variants from May 9 to the current day are deleted. Here are examples of logs from yesterday.
I also received the message - "There was an issue in the Cloud Billing cost management UI resulting in inaccurate data for May 8th (mostly 2pm to 8pm GMT). This issue was only fixed on May 8th and we are working on fixing this corrupted data. This issue does not affect BigQuery data export billing or invoicing."
@Roderick
I sent a private message to see if I can capture some more information from you and escalate this with our internal teams. Please share with me privately which customer you're with and if you have an open support instance. Thanks!