Getting Oath 2 access token issue

[sahushubham768]

Hi, 

We are fetching data from Big Query using Spring Boot and doing some processing in that data which is taking around 1-1.5 hours and after the process completes, we again have to hit the bigquery to fetch some more records but in between only we are getting 
Exception in thread "Thread-38" com.google.cloud.bigquery.BigQueryException: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential.

We are using the below approach to create the credential using the Access Token and providing the token expiry time as 2 hours.

Date oldDate = new Date(); 
Date newDate = new Date(oldDate.getTime() + TimeUnit.HOURS.toMillis(2)); 
Credentials credentials2 = GoogleCredentials.create(new AccessToken(token, newDate));

We have tried increasing the expiry time to 3 and 4 hours also but it is still failing with the same error.

When I checked the documentation to find from which method is throwing this issue, I found OAuth2Credentials class which has a method refreshAcessToken and this method is throwing that issue. OAuth2Credentials class is extending the Credentials class, still not able to find from where this error is coming as we are not using OAuth2Credentials 

One more fix I tried was creating OAuth2Credentials object reference instead of Credentials and overriding the refreshAccessToken method to provide my own implementation to refresh the Access token but that one is also not working. 
Can anyone please help me on this issue?

Could anyone help on this issue. What will be the more feasible way to approach this issue.

[ms4446]

The issue is that your BigQuery credentials are expiring prematurely, despite setting an expiration time of 2-4 hours. The error message, indicating that the OAuth2 access token is invalid, typically means it has either expired or been revoked. There are several factors could contribute to this problem:

• Token Revocation: Another process or part of your application might be inadvertently revoking the access token before it expires.
• Incorrect Token Refresh Logic: There might be an issue with how you're refreshing the access token, leading to it not being updated correctly.
• Clock Skew: A significant time difference between your application's system clock and Google's servers could cause premature token expiration.
• Network/Connectivity Issues: Transient network issues during token refresh could cause the process to fail.

Verify Token Revocation: Carefully review your code and any libraries to ensure no part of your application is inadvertently revoking the access token. Check your Google Cloud project's audit logs for any token revocation events.

Enhance Token Refresh Logic: Instead of manually creating GoogleCredentials, consider using the GoogleCredentialsProvider class, which handles token refresh more robustly:

GoogleCredentialsProvider credentialsProvider = FixedCredentialsProvider.create(
    ServiceAccountCredentials.fromStream(new FileInputStream("path/to/your/key.json"))
);
BigQuery bigQuery = BigQueryOptions.newBuilder()
    .setCredentials(credentialsProvider)
    .build()
    .getService();

If custom token refresh is necessary, ensure you're correctly implementing the OAuth2Credentials.refreshAccessToken() method.

Synchronize Clocks: Ensure your application server's clock is synchronized with a reliable time source (e.g., NTP). Even small clock skews can affect token validity.

Handle Network Issues: Implement retry logic with exponential backoff in your token refresh mechanism to handle transient network errors. Monitor your application logs for any network-related exceptions during token refresh.

Logging: Add detailed logging to your authentication and token refresh processes to help pinpoint when and why errors occur. Scopes: Ensure you're requesting the correct OAuth2 scopes for your BigQuery operations to avoid authorization issues. Credentials Management: Use a secure mechanism (e.g., environment variables, Kubernetes secrets) to store and manage your service account credentials. Avoid hardcoding them in your code.

GoogleCredentials credentials = ServiceAccountCredentials.fromStream(new FileInputStream("path/to/your/key.json"));

BigQuery bigquery = BigQueryOptions.newBuilder()
    .setCredentials(credentials)
    .setProjectId("your-project-id")
    .build()
    .getService();

Security: Protecting your service account credentials is crucial. Never expose them in public repositories or logs. Performance: For high-throughput applications, consider caching the BigQuery client instance to avoid excessive authentication overhead.

By following these steps, you can address the premature expiration of BigQuery credentials and ensure a more robust authentication mechanism for your application.

 

[sahushubham768]

Hi,

We are not using any key.json file to generate the Google Credentials instead of that we are using WIF integration token which further is used to generate the Access Token and using Access Token we are generating Google Credentials.
One thing that I want to highlight here is sometimes the application will work fine and not throw any issue suppose for example I have to process around 200k records at that time it will work fine but if records will be more than 300 or 350k means it will take more time to process and will fail with the error message.

• Token Revocation --> I checked the logs but didn't find anything related to other process or application trying to revoke the token.
• Incorrect Token Refresh Logic --> When I am overriding the OAuth2Credentials.refreshAccessToken() method I am generating the WIF integration token and Access Token again in that method and returning the new Access Token.
• Clock Skew --> My application is deployed in cloud run only and it is trying to access the same project big query views and some other project views.
• Network/Connectivity Issues --> I am not sure about network but as it is deployed in cloud run so it should not be a problem.

[ms4446]

The issue with your BigQuery credentials expiring prematurely is likely due to token refresh logic or network connectivity during long-running processes. The following example Overrides the OAuth2Credentials.refreshAccessToken()

import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.OAuth2Credentials;

public class CustomCredentials extends OAuth2Credentials {
    
    private String wifToken;
    
    public CustomCredentials(AccessToken accessToken, String wifToken) {
        super(accessToken);
        this.wifToken = wifToken;
    }

    @Override
    public AccessToken refreshAccessToken() throws IOException {
        // Logic to generate a new WIF token and then generate a new access token
        String newWifToken = generateWifToken();
        AccessToken newAccessToken = generateAccessToken(newWifToken);
        this.wifToken = newWifToken;
        return newAccessToken;
    }

    private String generateWifToken() {
        // Your logic to generate WIF token
    }

    private AccessToken generateAccessToken(String wifToken) {
        // Your logic to generate Access Token using WIF token
    }
}

In addition, you might want to implement a retry mechanism to handle transient network issues.

private <T> T executeWithRetry(Callable<T> callable) throws Exception {
    int maxAttempts = 5;
    int attempt = 0;
    while (attempt < maxAttempts) {
        try {
            return callable.call();
        } catch (Exception e) {
            attempt++;
            if (attempt >= maxAttempts) {
                throw e;
            }
            Thread.sleep((long) Math.pow(2, attempt) * 1000); // Exponential backoff
        }
    }
    return null;
}

 Add detailed logging around the token generation and refresh process. This will help you identify exactly when and why the token becomes invalid.

If possible, break down the processing of records into smaller batches to avoid long-running operations that might exceed token lifetimes.

By ensuring a token refresh implementation, adding retry logic, and optimizing record processing, you can mitigate these issues. Detailed monitoring and logging will also help in identifying and resolving any underlying problems.

[sahushubham768]

I have already tried extending the OAuth2Credentials and providing my own implementation for generating and refreshing token but that lead to a new error
Exception in thread "Thread-38" com.google.cloud.bigquery.BigQueryException: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential

I was generating the new token in constructor provided with the check if token is not expired then only generate new else use the same and as a result whenever I need Credentials anywhere in my project, I use to create the object reference of that class. I also have overridden the refreshAccessToken() method which generates a new AccessToken and getRequestMetadata() method to ensure the token is valid before each request. I have implemented the check for expiry in generation of new token at constructor and getRequestMetadata() both.

[ms4446]

It seems that the issue might be deeper in how the custom credentials class interacts with the BigQuery client and how token expiry and refresh are managed.You need try to streamline the implementation and ensure the credentials are managed correctly.

Ensure that the custom implementation of OAuth2Credentials correctly manages token refresh:

import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.OAuth2Credentials;

import java.io.IOException;
import java.util.Date;
import java.util.Map;

public class CustomCredentials extends OAuth2Credentials {

    private String wifToken;

    public CustomCredentials(String wifToken, AccessToken accessToken) {
        super(accessToken);
        this.wifToken = wifToken;
    }

    @Override
    public AccessToken refreshAccessToken() throws IOException {
        // Generate a new WIF token and then generate a new access token
        this.wifToken = generateWifToken();
        return generateAccessToken(this.wifToken);
    }

    private String generateWifToken() {
        // Your logic to generate WIF token
        return "newWifToken";
    }

    private AccessToken generateAccessToken(String wifToken) throws IOException {
        // Your logic to generate Access Token using WIF token
        Date newExpiryDate = new Date(System.currentTimeMillis() + TimeUnit.HOURS.toMillis(1)); // Set expiry time appropriately
        return new AccessToken("newAccessToken", newExpiryDate);
    }

    @Override
    public Map<String, List<String>> getRequestMetadata(URI uri) throws IOException {
        if (shouldRefresh()) {
            refresh();
        }
        return super.getRequestMetadata(uri);
    }

    private boolean shouldRefresh() {
        return getAccessToken().getExpirationTime().getTime() - System.currentTimeMillis() < TimeUnit.MINUTES.toMillis(5);
    }

    @Override
    public boolean hasRequestMetadata() {
        return true;
    }

    @Override
    public boolean hasRequestMetadataOnly() {
        return true;
    }
}

Next, create the BigQuery client using the custom credentials:

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;

public class BigQueryService {

    private BigQuery bigQuery;

    public BigQueryService(String wifToken) throws IOException {
        CustomCredentials customCredentials = new CustomCredentials(wifToken, generateInitialAccessToken(wifToken));
        this.bigQuery = BigQueryOptions.newBuilder()
            .setCredentials(customCredentials)
            .build()
            .getService();
    }

    private AccessToken generateInitialAccessToken(String wifToken) throws IOException {
        // Your logic to generate initial Access Token using WIF token
        Date newExpiryDate = new Date(System.currentTimeMillis() + TimeUnit.HOURS.toMillis(1)); // Set expiry time appropriately
        return new AccessToken("initialAccessToken", newExpiryDate);
    }

    public BigQuery getBigQuery() {
        return bigQuery;
    }
}

1. Ensure the refreshAccessToken() method correctly generates a new WIF token and access token.
2. Implement the expiry check in both the constructor and getRequestMetadata() method to ensure the token is refreshed before it expires.
3. Implement a retry mechanism in the application to handle transient errors.
4. Add detailed logging to track token generation, refresh, and usage. This can help identify issues in the process.

• Ensure environment variables and configurations are set correctly in Cloud Run.
•  If the application is multi-threaded, ensure thread-safety in your credentials handling.
• Thoroughly test the implementation with different record sizes and processing times to ensure stability.

public class Main {
    public static void main(String[] args) throws IOException {
        String wifToken = "yourInitialWifToken";
        BigQueryService bigQueryService = new BigQueryService(wifToken);
        BigQuery bigQuery = bigQueryService.getBigQuery();

        // Your BigQuery operations
    }
}

[Snoshone07]

Hi @sahushubham768 the issue you’re experiencing appears to be related to token expiration and how your implementation is managing credential refreshes. Here are some steps and recommendations to help resolve the problem:

1. Token Expiry and Auto-Refresh

• The AccessToken class you’re using doesn’t automatically refresh the token once it expires. Even if you manually set an expiry time, it won’t extend the token’s validity after it has expired.
• Instead, it’s better to use GoogleCredentials with the appropriate authentication scopes. This class is designed to handle token refresh automatically.

Example:

GoogleCredentials credentials = GoogleCredentials.getApplicationDefault() .createScoped(Collections.singletonList("https://www.googleapis.com/auth/cloud-platform")); BigQuery bigQuery = BigQueryOptions.newBuilder() .setCredentials(credentials) .build() .getService();

By using getApplicationDefault() or service account credentials, token refreshes are managed seamlessly by the Google API.

2. Use Service Account Credentials

If you’re not already using a service account, it’s highly recommended. Service accounts provide a secure way to authenticate and also handle token refresh automatically.

Steps:

1. Download the service account key (JSON file) from the Google Cloud Console.
2. Use the key in your application like this:

GoogleCredentials credentials = GoogleCredentials .fromStream(new FileInputStream("path-to-service-account-key.json")) .createScoped(Collections.singletonList("https://www.googleapis.com/auth/cloud-platform")); BigQuery bigQuery = BigQueryOptions.newBuilder() .setCredentials(credentials) .build() .getService();

3. Verify Token Refresh Logic

• If you’re overriding the refreshAccessToken method, ensure that your implementation correctly interacts with Google’s Identity Platform to obtain a new token.
• However, using Google’s official SDK makes this step unnecessary since the token refresh is handled internally.

4. Session Management During Long Processes

Given that your data processing takes 1–1.5 hours, the token might be expiring mid-execution. Using a GoogleCredentials instance ensures the token is refreshed as needed during your process.

5. Alternative Workflow Optimizations

If your process involves frequent BigQuery queries or handling large data volumes, consider restructuring the workflow:

• Use BigQuery batch processing to reduce the number of API calls.
• Export data to a storage bucket (e.g., Cloud Storage) and process it locally or in a distributed environment like Dataflow for improved efficiency.

6. Explore External Data Tools

For more complex workflows or prolonged processes, tools like Windsor.ai could simplify the integration. These tools securely fetch data from BigQuery and handle authentication, offering a streamlined approach for managing data workflows. Hope this helps