Google Merchant Transfers (GMC) Still Not Fully Working Several Days Since Setup

[Dan_Zrust]

Hi guys,

I am getting a little desparate here.  I've set up a new data transfers from my Merchant Center to BigQuery. I've opted for these tables:

1. Products
2. Price competitiveness
3. Best sellers

The only table with data after several days of running is "Products". The "Price competitiveness" and "Best sellers" still has zero rows.

In the transfer setup, there is this note:
The Market Insights data can’t be exported until you have enabled Market Insights in Google Merchant Center in "Growth > Manage programs". You must ensure that your use, or any third party's use, of the Market Insights data exported from your Google Merchant Center account complies with the Market Insights policy .

The note is nice but not relevant. The steps mentioned do not really prompt me to anything - I can e.g. already see all the price benchmarks in the web UI of my GMC so I believe I must have given the consent long time ago. 

So I am really wondering why the data from GMC is not coming through to BQ? Has anyone faced the same issue? 

I've already set up another transfer running for a different Merchant Center account from the same company with no issues so it's not the first time I am trying to set the transfer up.

[ms4446]

There are several potential reasons why the "Price competitiveness" and "Best sellers" tables might not be populating in BigQuery:

1. Data Availability: Market Insights data might not be available for all products. Products that aren't actively being sold or haven't garnered enough impressions or clicks might not generate Market Insights data.

2. Data Update Delays: Market Insights data may not be immediately updated in BigQuery. It can sometimes take up to 24 hours (or longer in some cases) for the data to reflect in BigQuery after being updated in the Merchant Center.

3. Transfer Issues: There could be issues with the data transfer process itself, preventing the data from being populated in BigQuery.

To address and potentially resolve the issue, consider the following steps:

• Transfer Status: Check the status of the data transfer in the Google Cloud Console. If there are any failures or errors, the provided messages might give insights into the root cause.

• Backfill Data Transfer: Consider running a backfill for the data transfer. This action forces the data transfer to execute for a specified date range, which might resolve any temporary glitches.

In the interim, remember that you can access both the "Price competitiveness" and "Best sellers" data directly from the Merchant Center web UI.

[Dan_Zrust]

@ms4446 
So the error I'm getting in BQ for the Price Competitivenes table is this:
Job 522587735028:merchant_center_653d692a-0000-2b74-9b23-94eb2c07575e (table Products_PriceBenchmarks_REDACTED$20231019) failed with error INVALID_ARGUMENT: No fields in the data are present in the table schema.; JobID: 522587735028:merchant_center_653d692a-0000-2b74-9b23-94eb2c07575e

=> And I would also assume that when I can see the Price Comp stats in the web UI, then I should be able to export these to BigQuery, no? 

[ms4446]

Yes, if you can view the Price Competitiveness data in the Merchant Center web UI, you should indeed be able to export it to BigQuery.

The error message "INVALID_ARGUMENT: No fields in the data are present in the table schema" suggests a mismatch between the data being transferred and the schema of the BigQuery table. This discrepancy can arise due to several reasons:

1. Schema Alterations: The schema of the BigQuery table might have been modified after setting up the data transfer.
2. Unexpected Data Format: The data being transferred from the Merchant Center might be in a format that differs from what the BigQuery table expects.
3. Transfer Issues: There could be inherent problems with the data transfer process itself.

To address this, consider the following steps:

• Schema Verification: Review the schema of the BigQuery table and ensure it aligns with the structure of the data being transferred from the Merchant Center.
• Sample Transfer: Initiate a sample data transfer with a limited dataset to verify if the transfer process is functioning correctly.

As an alternative solution, consider exporting the Price Competitiveness data to another format, such as CSV or JSON. Once you have the data in one of these formats, you can then manually import it into BigQuery.

[Dan_Zrust]

@ms4446 
I am not sure if you have played with the transfers yourself. 

But the way how it works is that when you set up the GMC=>BQ transfer, the schemas are autogenerated by the transfer itself. I am not specifiying any schemas as the end user.

And I have definitely not modified any of the automatically generated tables.

[ms4446]

Yes, the schemas for Google Merchant Center transfers to BigQuery are autogenerated by the transfer service.

Given this, the error message "INVALID_ARGUMENT: No fields in the data are present in the table schema" suggests an issue with the data transfer process itself.

You can try the following to address this:

1. Reinitialize the Transfer: Delete the current transfer and set up a new one. This action might prompt the system to regenerate the schema based on the most recent data format in the Merchant Center.

2. Backfill Data Transfer: Consider running a backfill for the data transfer. This forces the transfer to execute for a specific date range, potentially resolving any temporary glitches.

3. Google Cloud Support: If the issue persists, reaching out to Google Cloud Support might provide more specific insights and solutions.

[Dan_Zrust]

So I went ahead and removed the transfer an configured it again. 

Now I am getting the TopBrand,s TopProducts, Top_ProductsInventory tables which were previously failing.

But guess what... PriceBenchmarks continues to fail. Full error:
Job 522587735028:merchant_center_65369eba-0000-2dcc-a984-94eb2c073760 (table Products_PriceBenchmarks_REDACTED$20231020) failed with error INVALID_ARGUMENT: No fields in the data are present in the table schema.; JobID: 522587735028:merchant_center_65369eba-0000-2dcc-a984-94eb2c073760

[ms4446]

I am sorry to hear for that  you are still facing challenges with the Price Benchmarks table. It seems that there might be a specific issue with the data transfer process for this particular table. To delve deeper into the problem, consider the following steps:

1. Review Data Transfer Service Logs: Access the logs for the Data Transfer Service within the Google Cloud Console. These logs might shed light on the error's specifics and its potential origins.

2. Verify Data Format: Ensure that the Price Benchmarks data being transferred from the Merchant Center is consistent with the expected format and schema. Any inconsistencies or variations in the data might be the culprits behind the schema mismatch error.

3. Schema Integrity: Double-check the BigQuery schema for the Products_PriceBenchmarks_REDACTED table. Ensure that it hasn't undergone any manual modifications. If there have been changes to the schema, it's crucial to ensure that the incoming data aligns with this updated schema.

4. Google Cloud Support: If the problem remains unresolved, it would be beneficial to contact Google Cloud Support directly. Their expertise might help pinpoint issues with the Data Transfer Service or provide more specific troubleshooting steps.

[Dan_Zrust]

So again....

The tables are created by the transfer itself.... There is nothing to check. And I don't certainly edit the table schemas. 

I don't understand how I am supposed to check that "the Price Benchmarks data being transferred from the Merchant Center is consistent with the expected format and schema" when there is no data coming through. 

And I doubt there is any Google Cloud support for small projects like mine.

[SelaP]

Hi,

im facing the exactly same issue.

Whitout any change on my side the 

price_benchmark_timestamp that was a timestamp now needs to be a datatime.

I rename the autogenerated table to see if the new one created by the transfer job its working.

Now the process ends, new table is created and price_benchmark_timestamp is a datetime now but the content of the data is not correct.

price_benchmark_timestamp contains for instance 0024-02-01T20:34:25.498688 and this is not a valid date.

Thanks for your help.

[Dan_Zrust]

Bizarre. I got it to work and this was my process.

1. I already had a GMC transfer working with the price_benchmarks table being filled up.
2. I set up a new transfer for my problematic GMC account.
3. I deleted the autogenerated price_benchmarks schema (which had no columns anyway)
4. I copy the existing price_benchmarks table from the other GMC working transfer but I use the correct account suffix at the end of the name of my new table.
5. I wipe all existing rows from the table.
6. I re-ran the previously failed transfer for the new GMC account.
7. My copied table gets filled up.
8. Miracle. 

(Ofc, Google support here was useless, same as Google Merchant support... So I had to spend hours in figuring it out. But not really surprised. Google is very good in luring money from you and that's where it ends these days. Even this forum does not work properly and text just randomly disappears as I try to edit my response.) 

[Dan_Zrust]

OMG, I now see what you mean. Those geniuses has really changed the data type after I made it work via my process above.

I am now also getting across all my transfers:
Provided Schema does not match Table XXXXXX:google_merchant_center.Products_PriceBenchmarks_XXXX$20231025. Field price_benchmark_timestamp has changed type from TIMESTAMP to DATETIME

Well done Google...well done! Really trying hard to fed up everyone.

[SelaP]

Hi!

I still having the issue.

I removed the table (again) and re-execute the process and today instead of an invalid date the column price_benchmark_timestamp contains an "INVALID".

Would be great if we can have from google:

1- The reason behind to change a DDL contract

2- Where was announced

3- Steps to solve it

[SelaP]

Hi,

looks like Google fixed the issue because today the message is:

Provided Schema does not match Table Products_PriceBenchmarks_XXXXX. Field price_benchmark_timestamp has changed type from DATETIME to TIMESTAMP

So, they rollback and now the column is a Timestamp again.

The job is working again

[Dan_Zrust]

Yep, I submitted a bug via Google Cloud and they fixed it.