Hello Community,
I'm encountering a persistent and baffling error trying to access Firestore from a Python Flask application running on GKE Autopilot, and I've exhausted the standard troubleshooting steps I can think of. I'm hoping someone might have encountered something similar or can suggest other avenues to investigate.
Goal: Connect to our Firestore Native database (ID: (default), Region: us-central1) from a Python Flask app using the google-cloud-firestore library, authenticating via Workload Identity running on GKE Autopilot in us-central1. Project ID is hello-heylo.
Problem: All calls to Firestore (both read .get() and write .set() operations initiated via HTTP requests to the Flask app) consistently fail with the following error and traceback:
google.api_core.exceptions.NotFound: 404 The database (default) does not exist for project hello-heylo Please visit https://console.cloud.google.com/datastore/setup?project=hello-heylo to add a Cloud Datastore or Cloud Firestore database.
[2025-04-20 02:06:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2025-04-20 02:06:34 +0000] [1] [INFO] Using worker: sync
[2025-04-20 02:06:34 +0000] [7] [INFO] Booting worker with pid: 7
INFO:root:Firestore client initialized successfully (ADC, explicit database='(default)').
INFO:root:Storage client initialized successfully (ADC, implicit project).
INFO:root:Attempting to write document 'test_write_doc_v09' to collection 'gke_test_entries'...
ERROR:root:Error writing to Firestore: 404 The database (default) does not exist for project hello-heylo Please visit https://console.cloud.google.com/datastore/setup?project=hello-heylo to add a Cloud Datastore or Cloud Firestore database.
ERROR:root:--- Full Traceback ---
ERROR:root:Traceback (most recent call last):
File "/app/main.py", line 62, in test_firestore_write
doc_ref.set(test_data) # The actual write operation
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/cloud/firestore_v1/document.py", line 167, in set write_results = batch.commit(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/cloud/firestore_v1/batch.py", line 61, in commit commit_response = self._client._firestore_api.commit(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/cloud/firestore_v1/services/firestore/client.py", line 1430, in commit
response = rpc(
^^^^
File "/usr/local/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
return retry_target(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
_retry_error_helper(
File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
raise final_exc from source_exc
File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
result = target()
^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.NotFound: 404 The database (default) does not exist for project hello-heylo Please visit https://console.cloud.google.com/datastore/setup?project=hello-heylo to add a Cloud Datastore or Cloud Firestore database.
ERROR:root:----------------------
Notably, the Firestore client initialization itself (db = firestore.Client(...)) within the Flask app succeeds without errors. The error only occurs when an actual read/write operation is attempted.
- GKE Cluster: Autopilot, us-central1, Project ID hello-heylo.
- Application: Python 3.11 (python:3.11-slim base image), Flask 3.0.3, Gunicorn 22.0.0.
- Authentication: Workload Identity linking KSA hello-heylo-ns/hello-heylo-ksa to GSA hello-heylo-app-sa@hello-heylo.iam.gserviceaccount.com.
- Libraries (requirements.txt):
- Flask==3.0.3
gunicorn==22.0.0
Werkzeug==3.0.6
google-cloud-storage
google-cloud-firestore # Tried latest and pinned ==2.15.0
google-cloud-secret-manager
redis
# (pip, autopep8 also present)
(Note: google-cloud-storage operations work correctly from the same app).
What Has Been Verified (Checks Performed): I have confirmed the following configurations are correct:
- Firestore DB: The (default) database exists in project hello-heylo, is in Native Mode, and located in us-central1 (verified via Cloud Console screenshots).
- Manual Access: Reads and writes to this database work correctly when performed directly via the Cloud Console UI.
- API Enabled: The "Cloud Firestore API" is confirmed ENABLED for project hello-heylo (verified via Console screenshot).
- Billing: Project Billing is ACTIVE (verified via Console screenshot).
- GSA Permissions: The GSA hello-heylo-app-sa@... has the Project Editor role (verified via IAM page screenshot).
- Workload Identity Binding:
- The KSA hello-heylo-ns/hello-heylo-ksa is correctly annotated: iam.gke.io/gcp-service-account: hello-heylo-app-sa@... (verified via kubectl get sa -o yaml).
- The GSA hello-heylo-app-sa@... has the correct roles/iam.workloadIdentityUser binding for the KSA (serviceAccount:hello-heylo.svc.id.goog[hello-heylo-ns/hello-heylo-ksa]) (verified via gcloud iam get-iam-policy).
- Pod Identity: The running pod correctly receives the GSA identity. Verified via kubectl exec and querying the metadata server:
- Network Connectivity: Basic network connectivity from the pod to the Firestore endpoint works. Verified via kubectl exec and curl -v https://firestore.googleapis.com (shows successful DNS resolution, TCP connection, and TLS handshake).
- Client Code: The error occurs irrespective of client initialization:
- Implicit: firestore.Client()
- Explicit DB: firestore.Client(database="(default)")
- Client Library Version: The error occurs with both the latest (unpinned google-cloud-firestore) library and pinned version 2.15.0.
Relevant Code Snippet (main.py extract):
# Client Initialization (Example from v0.9/v0.10)
try:
db = firestore.Client(database="(default)") # Also tried without explicit DB
logging.info("Firestore client initialized successfully...")
except Exception as e:
logging.error(f"CRITICAL: Failed to initialize Firestore client: {e}")
# ...
# Example Failing Read Route (Write route fails identically)
@App.route('/test-firestore-read', methods=['GET'])
def test_firestore_read():
if db is None: return jsonify({"error": "Firestore client not initialized."}), 500
try:
collection_name = u'gke_test_entries'
doc_id = u'console_test_doc' # Document known to exist from UI test
doc_ref = db.collection(collection_name).document(doc_id)
logging.info(f"Attempting to read document '{doc_id}'...")
# --- THIS LINE FAILS with 404 DB Not Found ---
doc_snapshot = doc_ref.get()
# --- Code below is never reached ---
if doc_snapshot.exists:
logging.info(f"Successfully read document '{doc_id}'...")
return jsonify({"success": True, "data": doc_snapshot.to_dict()}), 200
else:
# ... doc not found handling ...
return jsonify({"success": False, "message": f"Document {doc_id} not found."}), 404
except Exception as e:
error_message = f"Error reading from Firestore: {e}"
full_traceback = traceback.format_exc()
logging.error(error_message)
logging.error("--- Full Traceback ---")
logging.error(full_traceback) # This shows the 404 DB not found error
logging.error("----------------------")
return jsonify({ "error": error_message, "exception_type": str(type(e)), }), 500
Question: Given that all standard configurations appear correct and have been meticulously verified, why would the Firestore backend persistently return 404 The database (default) does not exist for this service account identity when accessed via the client library from GKE Autopilot? Are there any other less common configurations, project settings, potential backend inconsistencies, or diagnostic steps I might be missing?
I'm unable to open a direct support case, so any insights or suggestions from the community would be greatly appreciated! Thank you!
LinkedIn
Twitter
