import Google play data to BigQuery

[mharitakis]

Hello,

I have managed to import Google play data using Data transfer service to Bigquery.

i noticed that from Google play there are reports from various years 2022,2021 etc but when i checked the tables in Bigquery there are data only from 2023.

Is there any way to verify that all reports have been imported to BigQuery? 

Thanks

[ms4446 · Accepted solution]

To ensure that all reports from Google Play have been successfully imported into BigQuery, follow these steps:

1. Check Transfer Logs:

   • Navigate to the Data Transfer page in the Google Cloud console.
   • Locate the transfer configuration for Google Play data.
   • Click on "View Logs" to examine the details of each transfer run. Pay attention to the success/failure status and the date ranges of the imported data. Look for any error messages or warnings that might indicate problems.

2. Review BigQuery Tables:

   • Inspect the BigQuery tables created by the data transfer.
   • Check the "Last modified" timestamp to determine when the tables were last updated.
   • Execute queries to verify the data, especially the range of dates. For example, use SELECT MIN(date_column), MAX(date_column) FROM your_table; to confirm that all expected years are present.

3. Inspect Transfer Configuration:

   • Review the transfer configuration to ensure it is set to import data from all the desired years.
   • Check for any date filters or restrictions that might limit the import scope. Ensure that the settings align with your data requirements.

4. Utilize BigQuery Audit Logs:

   • Access the BigQuery audit logs for comprehensive information about the import jobs.
   • Investigate these logs for any errors, warnings, or anomalies that might indicate issues with importing specific data ranges.

Additional Tips:

• Re-Running the Transfer: If you suspect that some data is missing, consider re-running the transfer for the missing date ranges.
• Cloud Monitoring: Implement Cloud Monitoring to track the transfer status. Set up alerts to notify you of any anomalies or issues during the data transfer process.
• Data Consistency Checks: Perform data consistency checks by comparing the record counts and types in BigQuery against the expected data from Google Play reports.