Maxwell Daemon and BigQuery

[RedCrick]

Is anyone here using Maxwell's Daemon to send MySQL CDC to BigQuery?

I have been trying to get it to work for a week now with very little luck.

Here is how I am running maxwell.

cd maxwell-1.40.0/bin
export GOOGLE_APPLICATION_CREDENTIALS=/home/red/redhistorydbprj001-468b7f03ed35.json
./maxwell --user=maxwell --password=$PW --producer=bigquery \
--filter='exclude: my_config.runtime_states, exclude: my_reporting.*' \
--log_level=trace \
--bigquery_project_id=redhistorydbprj001 \
--bigquery_dataset=new_data_set_01 \
--bigquery_table=examples

Then we I update a record in my `examples` table maxwell crashes ...

40586 [DEBUG] MysqlPositionStore: Writing binlog position to maxwell.positions: Position[BinlogPosition[0-0-1864034], lastHeartbeat=1680713765530], last heartbeat read: 1680713765530
40786 [DEBUG] C3P0PooledConnectionPool: Testing PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection@3e011b82] on CHECKOUT.
40787 [DEBUG] C3P0PooledConnectionPool: Test of PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection@3e011b82] on CHECKOUT has SUCCEEDED.
41536 [INFO] TaskManager: Stopping 3 tasks
41537 [ERROR] TaskManager: cause:
java.lang.IllegalArgumentException: JSONObject has fields unknown to BigQuery: root.database.
        at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:205) ~[google-cloud-bigquerystorage-2.14.2.jar:2.14.2]
        at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:164) ~[google-cloud-bigquerystorage-2.14.2.jar:2.14.2]
        at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:147) ~[google-cloud-bigquerystorage-2.14.2.jar:2.14.2]
        at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:106) ~[google-cloud-bigquerystorage-2.14.2.jar:2.14.2]
        at com.zendesk.maxwell.producer.MaxwellBigQueryProducerWorker.sendAsync(MaxwellBigQueryProducer.java:275) ~[maxwell-1.39.6.jar:1.39.6]
        at com.zendesk.maxwell.producer.AbstractAsyncProducer.push(AbstractAsyncProducer.java:93) ~[maxwell-1.39.6.jar:1.39.6]
        at com.zendesk.maxwell.producer.MaxwellBigQueryProducerWorker.run(MaxwellBigQueryProducer.java:252) ~[maxwell-1.39.6.jar:1.39.6]
        at java.lang.Thread.run(Thread.java:829) [?:?] 

Any help is appreciated. 

[RedCrick · Accepted solution]

I had failed to follow the documentation I found here: https://maxwells-daemon.io/producers/#google-cloud-pubsub 

Here's the `bq` command I used to correctly create my BigQuery table.

```

SCHEMA=database:string,table:string,type:string,ts:integer,xid:integer,xoffset:integer,commit:boolean,position:string,gtid:string,server_id:integer,primary_key:string,data:string,old:string
bq load --source_format=CSV new_data_set_01.tenants schema.sql $SCHEMA

```