This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

DataFlow Job Falied

Posted on 04-13-2023 12:23 AM
hminhas
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hi,

I am attempting to transform the data in the Dataprep tool. I import datasets from BigQuery using BigQuery External connection. After completing all the recipe steps when I run the output, it fails. I opened the ticket with the Trifacta team but got no solution.

Here you can find the issue detail discussed with the Trifacta support team
https://community.trifacta.com/s/question/0D5Do00000OE9dDKAT/getting-error-while-running-the-output-...

https://alteryx.my.site.com/CustomerCasePortal/s/customercases?id=5003n00002fApaTAAS

Error Message :

2023-04-03T13:48:16.604Z: Workflow failed. Causes: S05:PTableLoadTransformBigQuery/BigQueryIO.TypedRead/Read(BigQueryQuerySource)+PTableLoadTransformBigQuery/BigQueryIO.TypedRead/PassThroughThenCleanup/ParMultiDo(Identity)+PTableLoadTransformBigQuery/MapElements/Map+PTableLoadTransformBigQuery/ParDo(CaptureRowCount)+PMapTransform/ParDo(MapFunction)+PMapTransform2/ParDo(MapFunction)+PMapTransform3/ParDo(MapFunction)+PMapTransform4/ParDo(MapFunction)+PMapTransform6/ParDo(MapFunction)+PMapTransform7/ParDo(MapFunction)+PAggregateTransform/Combine.globally(Composed)/WithKeys/AddKeys/Map+PMapTransform8/ParDo(MapFunction)+PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey+PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/Combine.GroupedValues/Partial+PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey/Reify+PMapTransform9/ParDo(MapFunction)+PProfileTransform/CategoricalProfileTransform/ParDo(GetColumnValues)+PProfileTransform/PAggregateTransform/Combine.globally(Composed)/WithKeys/AddKeys/Map+PProfileTransform/CategoricalProfileTransform/Count.PerElement/Init/Map+PProfileTransform/CategoricalProfileTransform/Count.PerElement/Combine.perKey(Count)/GroupByKey+PProfileTransform/CategoricalProfileTransform/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+PProfileTransform/CategoricalProfileTransform/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+PProfileTransform/CategoricalProfileTransform/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write+PProfileTransform/PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey+PProfileTransform/PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/Combine.GroupedValues/Partial+PProfileTransform/PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey/Reify+PProfileTransform/PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey/Write+PAggregateTransform/Combine.globally(Composed)/Combine.perKey(Composed)/GroupByKey/Write+PTableLoadTransformBigQuery/BigQueryIO.TypedRead/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+PMapTransform5/ParDo(MapFunction)+PTableStoreTransformGCS/ParDo(CaptureRowCount)+PTableStoreTransformGCS/MapElements/Map+PTableStoreTransformGCS/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+PTableStoreTransformGCS/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+PTableStoreTransformGCS/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+PTableStoreTransformGCS/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+PTableStoreTransformGCS/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write failed., Internal Issue (562678cc9a33ec20): 63963027:24514

DataFlow Job Id:
2023-04-03_06_45_56-9435447789971286171

 

 

1 1 294
Topic Labels
1 REPLY 1

Posted on --/--/---- --:-- AM
Aris_O
Staff
Reply posted on --/--/---- --:-- AM

Hi @hminhas,

Welcome back to Google Cloud Community.

Based on the data and Dataflow Job ID you provided about the error you encountered. Here are some additional checking you may try while waiting for the Trifecta team's response.

  • Verify the BigQuery account's permissions before connecting to Dataprep. Make sure it has the right access rights to the BigQuery dataset and tables.

  • Verify that query you are using to extract the data from BigQuery to see if there are any problems. To check for issues, you can try running the query directly in BigQuery.

  • Verify that the BigQuery connector used by Dataprep is compatible with the BigQuery version you are utilizing.
  • Verify the firewall limitations or network connectivity issues that might be preventing the connection between Dataprep and BigQuery.

    You may follow up with the Trifacta team and include the DataFlow Job Id and the error message to help them investigate the issue further.