LinkedIn
Twitter
I have configured a BQ data transfer from Google Merchant Center. The transfer is scheduled to run every 6 hours but we observed from the "RUN HISTORY" of the transfer details is that following each scheduled run for the transfer, two more runs get scheduled:
- 30 seconds later for yesterday
- 60 seconds later for the day before yesterday
Both runs end up with jobs executing but they are logged in the history.
A run for example for 30.06.2024 which is scheduled on 01.07.2024 would show the following log details:
Where are these jobs coming from and what is their purpose exactly?
Thank you!
Solved
1 ACCEPTED SOLUTION
Hi @Mousa_Hamad,
Welcome to Google Cloud Community!
The possible cause of "two more runs getting scheduled" is the backfill feature of BigQuery Data Transfer Service. It detects missing data and retrieves data from the source to load into the BigQuery table. You can refer to managing backfill for more information.
Meanwhile, the job you're referring to is a load job, specifically transferring data from your source to BigQuery. One option available for batch loading in BigQuery is through BigQuery Data Transfer Service, which you are currently using to automate the transfer of data from Google Merchant Center to BigQuery.
BigQuery records the status of all jobs you submit:
- “Succeeded” status - a job completed successfully.
- “Failed” status - an error encountered.
In your situation, the message "summary: succeeded 0 jobs, failed 0 jobs" indicates that no jobs were executed during the time period you're reviewing.
Here are the possible reasons for this issue:
- The data was already completed before that timeframe.
- No data processing or analysis occurred during that specific period.
- No new data is available, or there are no updates.
- There might have been a temporary error; the transfer could have been paused or stopped.
- It might have been scheduled for a different time.
To troubleshoot:
- Review your scheduled transfer in the Data Transfer service on the left panel.
- Navigate to Data Transfer error logs and check the “Run History” or any encountered errors.
- Verify if your data source has new data for transfer.
- Lastly, ensure your transfer details are configured correctly.
For more information, you can also check Monitoring and Logging for BigQuery Data Transfer Service exports logs and metrics.
I hope the above information is helpful.
Hi @Mousa_Hamad,
Welcome to Google Cloud Community!
The possible cause of "two more runs getting scheduled" is the backfill feature of BigQuery Data Transfer Service. It detects missing data and retrieves data from the source to load into the BigQuery table. You can refer to managing backfill for more information.
Meanwhile, the job you're referring to is a load job, specifically transferring data from your source to BigQuery. One option available for batch loading in BigQuery is through BigQuery Data Transfer Service, which you are currently using to automate the transfer of data from Google Merchant Center to BigQuery.
BigQuery records the status of all jobs you submit:
- “Succeeded” status - a job completed successfully.
- “Failed” status - an error encountered.
In your situation, the message "summary: succeeded 0 jobs, failed 0 jobs" indicates that no jobs were executed during the time period you're reviewing.
Here are the possible reasons for this issue:
- The data was already completed before that timeframe.
- No data processing or analysis occurred during that specific period.
- No new data is available, or there are no updates.
- There might have been a temporary error; the transfer could have been paused or stopped.
- It might have been scheduled for a different time.
To troubleshoot:
- Review your scheduled transfer in the Data Transfer service on the left panel.
- Navigate to Data Transfer error logs and check the “Run History” or any encountered errors.
- Verify if your data source has new data for transfer.
- Lastly, ensure your transfer details are configured correctly.
For more information, you can also check Monitoring and Logging for BigQuery Data Transfer Service exports logs and metrics.
I hope the above information is helpful.
