Data Flow Create job from Template Error - MongoDB to Big Query

[mpetersonva]

I'm attempting to move some data in a MongoDB collection to Big Query using the MongoDB to Big Query Template.  The job runs sucessfully if a UDF is not setup.   However, if I add a Java Script UDF, the job failes with the following error.  Any suggestions?

{
"insertId": "3533598871770543158:674812:0:17463",
"jsonPayload": {
"line": "exec.go:66",
"message": "com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.\ncom.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@65a9ea3c. Client view of cluster state is {type=REPLICA_SET, servers=[{address=cluster0-shard-00-01.p6uep.gcp.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.SocketTimeoutException: connect timed out}}, {address=cluster0-shard-00-02.p6uep.gcp.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.SocketTimeoutException: connect timed out}}, {address=cluster0-shard-00-00.p6uep.gcp.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.SocketTimeoutException: connect timed out}}]\n\tat com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:408)\n\tat com.mongodb.internal.connection.BaseCluster.selectServer(BaseCluster.java:123)\n\tat com.mongodb.internal.connection.AbstractMultiServerCluster.selectServer(AbstractMultiServerCluster.java:54)\n\tat com.mongodb.client.internal.MongoClientDelegate.getConnectedClusterDescription(MongoClientDelegate.java:157)\n\tat com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:105)\n\tat com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.getClientSession(MongoClientDelegate.java:287)\n\tat com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191)\n\tat com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:211)\n\tat com.google.cloud.teleport.v2.mongodb.templates.MongoDbUtils.getMongoDbDocument(MongoDbUtils.java:112)\n\tat com.google.cloud.teleport.v2.mongodb.templates.MongoDbUtils.getTableFieldSchemaForUDF(MongoDbUtils.java:163)\n\tat com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.run(MongoDbToBigQuery.java:101)\n\tat com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.main(MongoDbToBigQuery.java:88)\n"
},
"resource": {
"type": "dataflow_step",
"labels": {
"project_id": "development-cloud-services",
"step_id": "",
"region": "us-east1",
"job_id": "2023-04-12_12_47_36-14855050917232087005",
"job_name": "locations01"
}
},
"timestamp": "2023-04-12T19:49:38.338997Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_type": "instance",
"compute.googleapis.com/resource_id": "3533598871770543158",
"dataflow.googleapis.com/job_id": "2023-04-12_12_47_36-14855050917232087005",
"compute.googleapis.com/resource_name": "launcher-2023041212473614855050917232087005",
"dataflow.googleapis.com/region": "us-east1",
"dataflow.googleapis.com/job_name": "locations01"
},
"logName": "projects/development-cloud-services/logs/dataflow.googleapis.com%2Flauncher",
"receiveTimestamp": "2023-04-12T19:49:48.156624963Z"
}

[Aris_O · Accepted solution]

Hi @mpetersonva,

Welcome Back to Google Cloud Community.

The error message indicates that the job failed due to a `MongoTimeoutException` This error occurs when the MongoDB driver cannot connect to the MongoDB server within the specified timeout period. In this case, the timeout period is set to 30 seconds.

Based on your investigation here are some possible answers for your come across errors:

• Verify that the MongoDB server is operational and reachable.
• Verify that the MongoDB URI and login information provided in the template are accurate.
• Verify that your connection settings, including the MongoDB replica set name, username, password, and, if necessary, the SSL certificate, are valid.
• If the MongoDB server is taking too long to respond, raise the timeout value. Setting the `socketTimeoutMS` and `connectTimeoutMS` parameters in the MongoDB connection string will enable you to achieve this.
• A firewall that is not configured properly can also be the source of the issue. Make sure that the ports needed by Big Query and MongoDB are open and available to the Dataflow job.
• Additionally, make that the login credentials used to access MongoDB and Big Query are valid and have the permissions needed to carry out the essential operations.
  
  Here are some references that might help you:
  
  https://cloud.google.com/dataflow/docs/guides/templates/provided/mongodb-change-stream-to-bigquery
  
  https://cloud.google.com/blog/products/databases/data-modernization-with-google-cloud-and-mongodb-at...
  
  https://cloud.google.com/dataflow/docs/guides/templates/provided/mongodb-to-bigquery