Error integrating Secret manager with Dataflow

[Pravish]

Apache Beam v.2.48
Secret Manager API v.2.22
Here's a simple code to fetch secrets from Secret Manager. When running with this on Dataflow, I run into an error (see below)

public class GCPSecretManagerClient implements Serializable {

    private static final Logger LOG = LoggerFactory.getLogger(GCPSecretManagerClient.class);

    private ProjectName projectName;
    private SecretManagerServiceClient secretManagerServiceClient;
    public GCPSecretManagerClient(String secretsProjectName) throws IOException {
        this.projectName = ProjectName.of(secretsProjectName);
        secretManagerServiceClient = SecretManagerServiceClient.create();
    }

    public String getSecret(String secretName) {

        GetSecretRequest request =
                GetSecretRequest.newBuilder()
                        .setName(SecretName.of(projectName.getProject(), secretName).toString())
                        .build();
        Secret response = secretManagerServiceClient.getSecret(request);
        return response.toString();

    }

    public void closeClient(){
        secretManagerServiceClient.close();
    }

}

 Error while creating secret manager client - 

exec.go:66","message":"java.lang.IllegalAccessError: class com.google.iam.v1.TestIamPermissionsRequest tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (com.google.iam.v1.TestIamPermissionsRequest and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app')
exec.go:66","message":"\tat com.google.iam.v1.TestIamPermissionsRequest.\u003cinit\u003e(TestIamPermissionsRequest.java:127)
exec.go:66","message":"\tat com.google.iam.v1.TestIamPermissionsRequest.\u003cclinit\u003e(TestIamPermissionsRequest.java:918)
exec.go:66","message":"\tat com.google.cloud.secretmanager.v1.stub.GrpcSecretManagerServiceStub.\u003cclinit\u003e(GrpcSecretManagerServiceStub.java:211)
exec.go:66","message":"\tat com.google.cloud.secretmanager.v1.stub.SecretManagerServiceStubSettings.createStub(SecretManagerServiceStubSettings.java:349)
exec.go:66","message":"\tat com.google.cloud.secretmanager.v1.SecretManagerServiceClient.\u003cinit\u003e(SecretManagerServiceClient.java:180)
exec.go:66","message":"\tat com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:162)
exec.go:66","message":"\tat com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:153)

[ms4446 · Accepted solution]

The error message you are seeing indicates that there is a problem with the Java class com.google.iam.v1.TestIamPermissionsRequest. This class is from the IAM library, but the error message does not necessarily mean that there is a problem with the IAM API itself. The error is more likely due to a Java class/method access issue caused by library conflicts.

Here are some things you can check to troubleshoot this issue:

• Make sure that all of your dependencies are using the same version of the IAM library.
• Make sure that there are not multiple versions of the IAM library in your classpath.
• Try using a different version of the IAM library.
• Try running your Dataflow job in a different region.
• Try using a different service account.

If you have checked all of the above and you are still getting the error, then please provide more information about your Dataflow job, such as the pipeline code and the steps that you are taking to run the job. I may be able to provide more specific help if I have more information.

Additional notes:

• The java.lang.IllegalAccessError is a Java runtime error that usually occurs when there is a version mismatch between compiled classes or libraries. It is not necessarily tied to permissions in the IAM sense.
• The mention of the Google IAM API v1 in the original error message is likely because the com.google.iam.v1.TestIamPermissionsRequest class is from the IAM library. However, this does not mean that the IAM API itself is causing the problem or that it is not enabled.
• The steps for checking permissions or the status of the IAM API are still relevant, but they should be considered separately from the java.lang.IllegalAccessError. IAM permission errors can also cause problems with Dataflow jobs, but they are typically manifested in different ways.

[Pravish]

The `google-cloud-secretmanager` dependency brought in it's own version of `grpc-protobuf-lite-1.55.1.jar`. This was overshadowed by another protobuf lib from `

beam-sdks-java-io-google-cloud-platform`. I excluded those and it worked for me.

Thank you!

[ms4446]

I'm glad to hear that you were able to resolve the issue! Dependency conflicts, especially with popular libraries like protobuf, can be tricky to debug, but identifying and managing those conflicts is key to ensuring the smooth operation of your application.

[saravanankumar1]

Hi Parvesh.
Please share the steps to resolve this issue . I'm facing the same situation .
Did you exclude these jars in the pom.xml file or in the class file during execution ? It would be nice to share the code base 

[saravanankumar1]

After excluding/including below class files in "shade" plugin of pom.xml build tag , the issue was resolved .Now I'm able to retrieve the secret while running the code in DataProc cluster invoked from Airflow DAG.

<configuration>

<relocations>

<relocation>

<pattern>com.google</pattern>

<shadedPattern>shaded.guava</shadedPattern>

<includes>

<include>com.google.**</include>

</includes>

<excludes>

<exclude>com.google.common.base.Optional</exclude>

<exclude>com.google.common.base.Absent</exclude>

<exclude>com.google.common.base.Present</exclude>

</excludes>

</relocation>

</relocations>

<filters>

<filter>

<artifact>*:*</artifact>

<excludes>

<exclude>META-INF/*.SF</exclude>

<exclude>META-INF/*.DSA</exclude>

<exclude>META-INF/*.RSA</exclude>

</excludes>

</filter>

</filters>

</configuration>