Unexpected Token Exchange Spikes in Secure Token API – Exceeding 20k Requests per Minute

[maad]

Hello Community,

I'm experiencing an unusual spike in Token Exchange per Minute for the Secure Token API in my Google Cloud project. Even though the number of active users is significantly lower than 20,000, the token exchange rate has exceeded this number at peak times.

Issue Details:

• Service: Secure Token API (securetoken.googleapis.com/secure_token_requests)
• Unexpected Behavior: Token Exchange per Minute exceeding 20,000 despite fewer active users.
• Quota Limits:
  • Token Exchange Per Minute: 18,000
  • Token Exchange Requests Per Minute: 6,000
  • Observed Peak Usage: Reaching 100% utilization intermittently.
• Error Trigger: This issue appears to happen when the app tries to store image data to Firebase Storage.

Questions for the Community:

• What could cause the Token Exchange per Minute to exceed the number of active users?
• Are there common misconfigurations in Firebase Auth or Firestore that could lead to excessive token requests?
• Could this be related to how Firebase handles authentication refresh tokens when accessing Storage?
• Any best practices to optimize token exchange usage and avoid unnecessary authentication requests?

I’d appreciate any insights or troubleshooting suggestions! I’ve attached relevant screenshots for context.

Thanks in advance!

[KyleMari]

Hi @maad,

Welcome to Google Cloud Community!

I’ve provided my answers to each of the key questions you have regarding the Token Exchange per Minute usage.

Q: What could cause the Token Exchange per Minute to exceed the number of active users?

Answer:

• Token expiration and reauthentication: Based on the documentation, Firebase ID tokens are short lived. With the help of a refresh token, new ID tokens are being granted to renew a user’s login session. With that said, it can be a factor if the refresh logic in your code is somewhat misconfigured, forcing the client app to request for new tokens even if the existing one isn’t expired yet.
• Frequent uploads or retries upon failure: If one user frequently uploads images to Cloud Storage, even if there are only around 2000 users in total, it’s best to observe by average how many images are being uploaded by every user within any given timeframe. The same thing goes for failed uploads (like due to network failures). If your app is reattempting to gain access many times, this can contribute to metric spike.
• Background processes in action: Even if your user isn’t directly interacting with the app, could it be the case that the app is implemented in such a way that certain asynchronous tasks are taking place that involve Cloud Storage?

Q: Are there common misconfigurations in Firebase Auth or Firestore that could lead to excessive token requests?

Answer:

• Excessive token refresh: Just like the first answer in the above question, if your app has certain functionalities that manually trigger token refresh or reauthentication more than necessary, this can lead to excessive token requests.
• Increased operations that require authentication: Also, even though Firestore doesn’t directly deal with token exchanges, there’s a chance that this may be due to broad security rules. The broader the rules are implemented, the wider the chance for data access to be possible. Apart from security rules, numerous operations that require a user to undergo authentication in each try can add up to the metric (e.g. excessive back and forth operation between logging in and logging out).

Q: Could this be related to how Firebase handles authentication refresh tokens when accessing Storage?

Answer: Not necessarily when it comes to accessing Storage but it can be a factor. If the app ends up attempting to refresh tokens or requiring a user to reauthenticate more than necessary, they can be part of the problem.

Q: Any best practices to optimize token exchange usage and avoid unnecessary authentication requests?

Answer: Though the insights that I do have may only be a limited set of tips (as I may only have limited visibility on the issue that you’re encountering), you can consider the following methods:

• Make sure that reauthentication attempts are properly implemented in your app: Since reauthentication may always increase the token exchange count, make sure that this isn’t what’s causing the issue. Perhaps, you can use listeners to properly track the status of your Auth instance. You can also check out the following references for more insights:
  • Users in Firebase Project - Firebase Auth
  • Why is it a bad idea to force refresh access token - Stack Overflow
• Consider implementing the best strategy for upload retries (if applicable): If and only if you’re using this mechanism, also make sure that there’s a reasonable length of time for a delay during retry attempts in case a user fails to upload an image.
• Review security rules implemented for Firestore and even for Cloud Storage.
• Continuously monitor the API usage: See if each and every improvement that you can make for your app makes progress towards proper resolution of your spiking issue.

If the issue persists, I would recommend reaching out to Google Cloud support or Firebase support if the quota issue that you encountered needs further investigation. Though I’m hopeful that the initial troubleshooting steps and optimization tips would help, there could be other underlying issues that need to be examined further.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.