Persistent "API Key Not Valid" Error on Cloud Run Deployment (Gemini API)

[tedbanks]

• Description: Despite successful local application functionality and direct curl tests of the Gemini API (using a valid API key), our Cloud Run service consistently reports "API key not valid" errors from the Generative Language API. This prevents AI insight generation in the deployed environment.
• Key Findings & Steps Taken:
  • Local Success: The application (React frontend, Node.js/Express backend proxying Gemini API) functions perfectly locally in Cloud Shell, with AI insights generating as expected.
  • Backend Confirmed Key Loading: Debugging logs (console.log('DEBUG: GEMINI_API_KEY value:', GEMINI_API_KEY);) within the deployed Cloud Run service show that the GEMINI_API_KEY environment variable is correctly populated with the full API key from Secret Manager.
  • Direct curl Test Success: A curl command using the exact same API key and gemini-1.5-flash model, directly to the generativelanguage.googleapis.com endpoint, successfully returns AI content.
  • Secret Manager Configuration: The gemini-api-key secret is correctly created in Secret Manager, and the Cloud Run service account has the roles/secretmanager.secretAccessor role.
  • Model Name Confirmation: The gemini-1.5-flash model is used, which was confirmed available and working via curl.
  • Region Agnostic: Deployment attempts in both us-central1 and us-west1 regions exhibit the same "API key not valid" error.
  • Base64 Encoding Attempt: To rule out hidden character issues, the API key was Base64-encoded in Secret Manager, and the backend was updated to decode it. This resulted in a new TypeError: Cannot convert argument to a ByteString... within the container, suggesting corruption after Secret Manager injection but before decoding or API call preparation.
  • Observed Discrepancy: The API key not valid error is received from Google's Generative Language API only when the call originates from the Cloud Run container, never from local execution or direct curl.