DataFlow job failed with Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Requested deidentify

[CristianMihai]

Hello, 

Following an official video from GCP on YouTube (https://www.youtube.com/watch?v=Kc7_wDDctAc&list=PLIivdWyY5sqK9j4_JkC8j1mY4JEGgLdcD&index=6) I am trying to replicate Masking from CSV and write in BigQuery with a Cryptographic key but my DataFlow job throwing many of these errors even if the deidentify template is right configured and the path it`s also correct. The only difference is that I can't do it globally so I am done in the same setting in europe region.

2023-11-09 12:48:13.870 EET

Error message from worker: com.google.api.gax.rpc.NotFoundException: io.grpc.StatusRuntimeException: NOT_FOUND: Requested deidentify template not found. com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:90) com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41) com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86) com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66) com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84) com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127) com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286) com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055) com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807) io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:578) io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:548) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:814) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:494) io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Requested deidentify template not found. io.grpc.Status.asRuntimeException(Status.java:539) io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:548) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:814) io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:494) io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.java:833)

[Poala_Tenorio]

Make sure that the deidentification template you are referring to exists in the specified region (Europe in your case) within the Cloud Data Loss Prevention (DLP) service. Double-check the name, permissions, and availability of the template since the error message indicates that the requested deidentification template is not found. This issue commonly arises when the DataFlow job is unable to locate or access the specific deidentification template needed for the masking process.

Confirm that the service account or identity used by your DataFlow job has the necessary permissions to access the deidentification template in the specified region. This includes both read and use permissions on the DLP templates. Also check if the service account used by your DataFlow job has appropriate permissions not just for DataFlow but also for accessing DLP templates. Consider granting the "roles/dlp.user" or more specific DLP-related roles to the service account if needed.

Sometimes, cross-region access can cause issues with template retrieval. Verify that the DataFlow job and the DLP deidentification template are in the same region.

Explore the logs in the Google Cloud Console for more detailed error messages or any additional information that might pinpoint the issue. Sometimes, more specific error codes or messages can provide insights into the root cause.

[CristianMihai]

Hello Poala_Tenorio, 

Thank you for your answer. 

I already checked all necessary rights and regions to be the same. I also tried one in locations/europe and one in Belgium (europe-west1). Same error for both tries. 

Meantime one of your colleagues Murat Eken answered me with this: 

Hi Cristian,

That's unfortunately a known limitation (see the docs here: https://cloud.google.com/dataflow/docs/guides/templates/provided/dlp-text-to-bigquery)

"This template does not support a regional path for de-identification template location. Only a global path is supported.

The template itself is in preview, I'll try to verify what the plans are, but in the meantime, you can create a custom Dataflow job yourself and implement the functionality (using Python/Java)."

I can create a custom Dataflow but i dont know how to implement same functionality securly in Python and there is no documentation of that(i think). 

So is this true, we cant use de-identification template other than on global, or not?

Waiting for your answer!