Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Data loss in Big Query while using Firebase.

We are seeing that the metric numbers that we are calculating using BQ are coming out to be very low. One of the reasons that they can think of is "Data not being transferred to BQ by Firebase".

Can I know the RCA of the issue.

0 3 491
3 REPLIES 3

Here are some steps and considerations to help you troubleshoot and resolve this issue:

 

  • Verify Firebase-BigQuery Integration:

    • Integration Status: In your Firebase console, navigate to the "Integrations" page and confirm that BigQuery is listed as "Linked". If not, follow the provided instructions to establish the link.
    • Permissions: Check that the Firebase service account has the necessary BigQuery permissions. The required roles are typically roles/bigquery.dataEditor and roles/bigquery.jobUser.
  • Check Export Settings for Firebase Products:

    • Supported Products: Confirm that data export is enabled for relevant Firebase products such as Google Analytics, Crashlytics, Performance Monitoring, Cloud Messaging (FCM), and A/B Testing (note that A/B Testing has limited data export).
    • Export Activation: Within your Firebase console, ensure that data export to BigQuery is enabled for each product you're using (e.g., Analytics settings).
  • Investigate BigQuery Datasets:

    • Correct Datasets: Verify that you're querying the correct datasets in BigQuery, typically named analytics_XXXXXX, where "XXXXXX" corresponds to your project identifier.
    • Data Freshness: Remember that Firebase exports data to BigQuery daily. Intraday export is available for some products, but generally, expect a delay from data generation to its availability in BigQuery.
  • Troubleshooting with BigQuery Transfer Service:

    • Service Monitoring: In the Google Cloud Console, check the BigQuery Transfer Service for any error messages, transfer run statuses, and configuration issues that might indicate problems in the data transfer process.
  • Common Pitfalls to Avoid:

    • Data Filtering: Review your BigQuery queries to ensure no inadvertent data exclusion due to filters.
    • Time Zone Alignment: Align timestamps between Firebase and BigQuery, considering potential time zone differences for accurate calculations.
    • Sampling in Analytics: If using Google Analytics data, be mindful of data sampling, especially in non-Google Analytics 360 accounts, which could affect the accuracy of your reports or queries.

 

Customers have set up google analytics to bigquery export .But the customer is seen comparing to the other way they stored the app users details are mismatching.

Earlier there was no mismatch past few days we see there is lots of data is missing

Hey Nishin, 
Were you able to resolve this issue? If yes, could you provide some details on the same?