INTERNAL ERRORS Bigquery Data Transfer for Salesforce

[arohadi]

I have recently encountered the following errors:

Transfer run failed due to internal errors. Please try again later.

The thing is; we have multiple salesforce tables to be extracted, many have succeed. All of which have the same settings and configurations in the Bigquery data transfer, yet somehow, these particular table(s) have been constantly failing due to internal errors.

The salesforce API has also been configured properly as it works through the other platform.

Unfortuntately, there's no specific error cause in the bigquery data transfer log.

[anjalisoni3655]

Can you please specify what errors are you facing?

[arohadi]

I'm afraid there's not much detail about it. Here's the log:

{
"insertId": "1gqekf7dswxb",
"jsonPayload": {
"message": "Transfer run failed due to internal errors. Please try again later."
},
"resource": {
"type": "bigquery_dts_config",
"labels": {
"config_id": "6625d7c8-0000-21a9-9d1e-582429a8e28c",
"location": "us",
"project_id": ...
}
},
"timestamp": "2024-05-02T16:47:09.220870Z",
"severity": "ERROR",
"labels": {
"run_id": "6656b54b-0000-27ca-aefc-14c14ef15a90"
},
"logName": "projects/.../logs/bigquerydatatransfer.googleapis.com%2Ftransfer_config",
"receiveTimestamp": "2024-05-02T16:47:09.902484621Z"
}

[anveshc05]

Hi, This question seems to be in the wrong forum. Can you check if it is related to Integration Connectors which is added as a topic label? 
If this is related to data transfer , can you ask this question with the Data transfer label in the correct forum.

[DariotraSec]

Hello guys! I am getting the same error in the bigquery transfer for Salesforce. Did you were able to fix your issue? 

[Snoshone07]

Hi @arohadi I understand how frustrating it can be to have some Salesforce tables transfer successfully to BigQuery while others fail, even with identical configurations. Here’s a step-by-step approach to help you resolve the issue:

1. Check Data and Schema for Failing Tables

The issue could stem from inconsistencies in the data or schema of the failing tables.

Verify Table Schema:

• Compare the schema of the failing tables with those that transfer successfully.
• Look for any differences, such as unsupported data types or fields with excessively large sizes.

Inspect for Data Anomalies:

• Review the records in the failing tables for unusual or corrupted data that might cause errors during the transfer.

2. Review API Query Limits and Configurations

While the Salesforce API may be functioning correctly elsewhere, BigQuery’s Data Transfer Service could still encounter limitations.

Check Salesforce API Usage Limits:

• Ensure you’re not exceeding Salesforce’s API query limits during the transfer process, as this could lead to intermittent failures.

Consider Large Objects or Custom Fields:

• If the failing tables include large objects (like attachments) or numerous custom fields, confirm that the API configuration is capable of handling them.

3. Analyze Logs and Retry Transfers

Logs can provide valuable insights, even if error messages appear vague.

Enable Debugging Logs:

• Enable debugging for your BigQuery data transfer to gather more detailed error logs.
• Check Salesforce API logs to see if any errors are reported on their end during the transfer.

Retry Transfers:

• Attempt to manually retry the transfer at different times to rule out temporary service interruptions or API rate limitations.

4. Consider an Alternative Integration Tool

If the problem persists, using a tool like Windsor.ai might be a more effective solution for transferring Salesforce data to BigQuery.

Why Windsor.ai?

• Seamless integration with BigQuery, handling complex schemas and transformations automatically.
• Enhanced error reporting, making troubleshooting more straightforward.
• Reliable performance, especially for tables with custom fields or large objects.

Greetings and good luck !